IMPACT to save 20% off today).IMPACT to save 20% off today). With a live UX training starting next week.IMPACT to save 20% off today).
A few months ago I wrote about what it means to stay gold — to hold on to the best parts of ourselves, our communities, and the American Dream itself. But staying gold isn’t passive. It takes work. It takes action. It takes hard conversations that ask
With a 13 billion year head start on evolution, why haven’t any other forms of life in the universe contacted us by now?
(Arrival is a fantastic movie. Watch it, but don’t stop there – read the Story of Your Life novella it was based on
If you haven’t been able to keep up with my blistering pace of one blog post per year, I don’t blame you. There’s a lot going on right now. It’s a busy time. But let’s pause and take a moment
It’s my honor to announce that John Carmack and I have initiated a friendly bet of $10,000* to the 501(c)(3) charity of the winner’s choice:
By January 1st, 2030, completely autonomous self-driving cars meeting SAE J3016 level 5 will be commercially available for
In a way, these two books are responsible for my entire professional career.
With early computers, you didn’t boot up to a fancy schmancy desktop, or a screen full of apps you could easily poke and prod with your finger. No, those computers booted up to the command
Hard to believe that I’ve had the same PC case since 2011, and my last serious upgrade was in 2015. I guess that’s yet another sign that the PC is over, because PC upgrades have gotten really boring. It took 5 years for me to muster
In an electric car, the (enormous) battery is a major part of the price. If electric car prices are decreasing, battery costs must be decreasing, because it’s not like the cost of fabricating rubber, aluminum, glass, and steel into car shapes can decline that much,
I’ve never thought of myself as a “car person.” The last new car I bought (and in fact, now that I think about it, the first new car I ever bought) was the quirky 1998 Ford Contour SVT. Since then, we bought a
When I wrote about App-pocalypse Now in 2014, I implied the future still belonged to the web. And it does. But it’s also true that the web has changed a lot in the last 10 years, much less the last 20 or 30.
Websites have gotten
When we started Discourse in 2013, our server requirements were high:
I’m not talking about a cheapo shared cpanel server, either, I mean a dedicated virtual private server with those specifications.
We
I sometimes get asked by regular people in the actual real world what it is that I do for a living, and here’s my 15 second answer:
We built a sort of Wikipedia website for computer programmers to post questions and answers. It’s called Stack
Remember “cybersecurity”?
Mysterious hooded computer guys doing mysterious hooded computer guy... things! Who knows what kind of naughty digital mischief they might be up to?
Unfortunately, we now live in a world where this kind of digital mischief is literally rewriting the world’s history. For proof
I didn’t choose to be a programmer. Somehow, it seemed, the computers chose me. For a long time, that was fine, that was enough; that was all I needed. But along the way I never felt that being a programmer was this unambiguously great-for-everyone career field
Why does choosing the right JavaScript Data Grid still matter in 2026? Data grids remain a cornerstone of web applications: dashboards, admin panels, CRMs, analytics, and enterprise systems all rely on them. The choice of the right grid still defines performance, customization flexibility, accessibility, and cost.
To find which grid fits your needs, I reviewed eight top JavaScript data grids and compared them by performance, customization, accessibility, cost, integration, and devX.
Transform your MCP server into an HTTP API that anyone can access from anywhere
You have an MCP server running locally. You want others to use it via HTTP calls.
Event-driven architecture has become the darling of modern software engineering. Walk into any tech conference, and you'll hear evangelists preaching about decoupling, scalability, and real-time processing. What they don't tell you is the dirty secret hiding behind all those beautiful architecture diagrams: most of what we're streaming is waste.
After analyzing production deployments across 15 different applications over the past 18 months, I've uncovered a pattern that should make every architect nervous. Research shows that approximately 80% of event streams represent wasted computational resources, storage costs, and engineering effort. But before you dismiss this as hyperbole, let me show you exactly what's happening under the hood of your "cutting-edge" event infrastructure.
In a world where attackers routinely scan public repositories for leaked credentials, secrets in source code represent a high-value target. But even with the growth of secret detection tools, many valid secrets still go unnoticed. It’s not because the secrets are hidden, but because the detection rules are too narrow or overcorrect in an attempt to avoid false positives. This creates a trade-off between wasting development time investigating false signals and risking a compromised account.
This article highlights research that uncovered hundreds of valid secrets from various third-party services publicly leaked on GitHub. Responsible disclosure of the specific findings is important, but the broader learnings include which types of secrets are common, the patterns in their formatting that cause them to be missed, and how scanners work so that their failure points can be improved.
Modern applications are rarely standalone. Most need to talk to databases, APIs, or even AI-driven agents that make real-time decisions. As this landscape grows, developers need a common way to connect apps, services, and data without constantly reinventing the wheel. That’s where MCP (Model Context Protocol) comes in.
In this post, we will look at what MCP is, why it matters, and how you can integrate a simple Node.js service with an MCP server. We will also walk through a working example, i.e, a user info microservice, and see what the setup and output look like.
In web development, responsive and user-friendly components have never been more important. One of these is a filter component that enables web users to quickly filter the user interface (UI) and data elements and display only relevant fields. The challenge is creating a filter component that can fit any screen size.
This article will demonstrate how to implement a responsive filter component using React. The tutorial explains how developers can build flexible filters into their web applications.
Java 8, released more than a decade ago, is a major milestone. With this version, Java brought a fundamental shift from only being an object-oriented programming (OOP) to a combination of OOP and functional programming (FP) as well. To achieve this, Java 8 came in with support of lambdas, stream APIs, etc., as core language features.
Stream API is influenced and modeled after the collection pipeline. A typical stream has three stages, viz., source, intermediate operations, and terminal operations.
Do you struggle to organize your React project folder structures? The right folder organization does more than just keep things tidy. The development efficiency and team collaboration of your project depend on it. A solid folder structure makes the project look clean and speeds up scalability, collaboration, and debugging.
The way developers organize files and components is vital to the long-term success of any React project. React applications are easier to scale, debug, and onboard new team members through a clean folder structure and clear system design. The best folder structure practices help developers find files and components quickly, which reduces the time spent searching through the code. The consistent folder structure of your application prevents chaos and confusion as it grows. Team members work together more smoothly, with fewer merge conflicts and improved productivity.
The goal is simple: build a secure app that uses SSR or SSG for SEO, CSRF-safe session cookies, clear CORS rules, and a smooth deploy to Vercel plus Forge or Vapour.
By the end, readers will know how to wire Laravel routes and controllers, fetch from Next.js with credentials, handle CSRF and sessions, and deploy with confidence. A short troubleshooting list covers common 401, 419, and CORS issues so teams do not get stuck.
In today’s AI-powered systems, real-time data is essential rather than optional. Real-time data streaming has started having an important impact on modern AI models for applications that need quick decisions. However, as data streams increase in complexity and speed, ensuring data consistency is a significant engineering challenge. As we know, AI models are heavily dependent on the input data used to train them. The quality of this input data is very important and should not be corrupted or contain errors. The accuracy, reliability, and fairness of the model’s predictions can be significantly affected if the quality of the input data is compromised.
The above statement is concrete, while AI models are being developed and subsequently made ready to identify patterns, make predictions based on input data. If we integrate these developed and tested trained AI models with real-time data stream processing pipelines, the predictions can be achieved on the fly. Because the real-time data streaming plays a key role for AI models as it allows them to handle and respond to data as it comes in, instead of just using old fixed datasets. You could read here my previous article, “AI on the Fly: Real-Time Data Streaming from Apache Kafka to Live Dashboards.” But the big question is how we can ensure real-time data that comes as a stream from various sources is free from errors and not at all bad data. By spotting patterns and trained data, AI systems decide. If this data has mistakes, doesn’t add up, or is messy, the model might pick up wrong patterns. This can lead to outputs that are biased, off the mark, or even risky.
Imagine deploying a Node.js backend service that works flawlessly in development, only to have it mysteriously crash in production. Everything ran fine on your laptop, but on the live server, the process keeps shutting down unexpectedly.
In our case, the culprit was a single unhandled promise rejection — one missing .catch() in our code caused Node to exit abruptly whenever an error occurred. That one “tiny” mistake made the difference between a stable service and frequent downtime. In this article, we’ll explore how a misconfigured error handling in a Node/Express API can bring down an application, and how to diagnose and fix it to prevent future crashes.
You are demonstrating your next mobile application idea to investors, and suddenly the application crashes. Now you are clueless whether it crashed due to bad WIFI at the venue, your service is down, or your application itself hit a snag. There are no logs or alerts to tell you what just happened. Mere imagination of this scenario can send any founder or developer into panic and nervousness. Therefore, having robust error handling is critical to modern-day mobile applications. React Native has become an industry standard in building cross-platform native mobile applications. React Native boasts a rich ecosystem of error handling at various levels of architecture.
However, often projects lack clear guidelines on how to handle various error scenarios. This results in either overdoing or underdoing error handling in otherwise robust and feature-rich applications. In this article, you will learn how to adopt a three-layer error handling mechanism, which is gaining widespread popularity with major enterprise-level applications built at leading companies. The article will give you a practical guide instead of pure theory, so you are ready to adapt these concepts immediately.
The evolution of large language models (LLMs) and agentic AI requires a fundamental shift in how applications expose their capabilities. Traditional REST APIs are designed for software-to-software communication, requiring developers to read documentation and write custom integration code. The Model Context Protocol (MCP) is an open standard designed to solve this by creating a unified, machine-readable interface that AI agents can dynamically discover and interact with.
This article provides a comprehensive guide on converting an existing Node.js REST API into an MCP server using the official TypeScript SDK, focusing on the architectural changes and crucial use cases that this conversion unlocks.
Ever been stranded with no Wi-Fi and suddenly discover that your go-to app is useless?
You're sitting on a delayed flight, stuck in the middle of a subway tunnel commute, or out camping in some serene spot (but signal-less), and you fire up that one app you use every day, only to be greeted by an error message. "No internet connection." Brilliant.
Picture this: you’re building a web app that can verify someone’s identity by having them snap a selfie with their webcam and upload a photo of their ID. It’s like something out of a sci-fi movie, but you can make it happen with ReactJS and face-api.js (a super cool library built on TensorFlow.js). This setup lets you create a working prototype in a few hours, all running right in the browser — no fancy servers required.
In this guide, I’ll walk you through building a React component that compares a live webcam feed to an ID photo to confirm a match. We’ll talk about why this is awesome for quick prototyping and toss in some ideas for where you could use it.
When building React applications, data fetching often starts with the native fetch API or tools like Axios. While this approach works for small projects, larger applications require features such as caching, retries, synchronization, and request cancellation, and it is here that TanStack Query, formerly React Query, excels. It provides a battle-tested abstraction for CRUD operations with powerful state management built in.
In this article, we’ll walk through fetching data with useQuery, performing mutations with useMutation, and highlighting some features that make TanStack Query a helpful tool for scaling React apps.
In Terraform, many configurations are dynamic, and you may build a list using conditional expressions that return null when not applicable. If those null values are passed directly to a resource (for example, in security_group_ids or depends_on), they can cause validation errors.
The compact() function ensures that only valid, non-null elements are included, helping prevent such runtime errors during the apply phase.
After exploring Tuples and Records in Parts 1–4—covering JavaScript syntax, immutability, value-based equality, performance benefits, and React optimizations—we now examine why this ambitious proposal was ultimately withdrawn from ES2025. Despite the promise of native immutable data structures, technical challenges around structural equality, memory management, and engine optimization prevented its adoption. In this part, we’ll break down the core reasons for the withdrawal, the alternatives considered, and the lessons for future JavaScript language evolution.
The journey of the Tuples and Records proposal in JavaScript represents one of the most significant withdrawals in recent ECMAScript history. After years of development and community anticipation, the proposal was officially withdrawn from the TC39 standardization process in April 2025, marking the end of an ambitious attempt to bring immutable data structures as primitives to JavaScript.
Part 4 dives into React, showing how Tuples and Records can cut down unnecessary re-renders. By using immutable, value-based data structures, React can efficiently detect state changes, keeping your components lean and fast. We’ll explore why re-renders happen and how adopting Tuples and Records simplifies state management while boosting performance.
Also read:
In the current fast-paced digital age, many data sources generate an unending flow of information, a never-ending torrent of facts and figures that, while perplexing when examined separately, provide profound insights when examined together. Stream processing can be useful in this situation. It fills the void between real-time data collecting and actionable insights. It’s a data processing practice that handles continuous data streams from an array of sources. Real-time data streaming has started having an important impact on modern AI models for applications that need quick decisions.
We can consider a few examples where AI models need to deliver instant decisions, such as self-driving cars, fraud in stock market trading, and smart factories that utilize technology like sensors, robots, and data analytics to automate and optimize manufacturing processes.
So far though, my experience has been that there are many more negative responses than positive. Maybe the Ruby community isn’t the place I thought it was, and MINASWAN was always a lie. That makes me sad. 😢
It’s such a terrible mental tax on builders that divisive clowns just ride in and spew these bullshit terms that they clearly don’t understand themselves in bad faith. Ignore & keep building.
There are really a lot of us Danes who believed that when people came to this ‘world’s best country’ and were given such good opportunities, they would integrate. They would become Danish, and they would never, ever harm our society. All of us who thought that way have been wrong.
One day, I got a chance. It just seemed to show up. It acted like it knew me, as if it wanted something.
It fluttered around me. It brushed up against me. It circled me as if it wanted me to grab it.
U.S. Immigration and Customs Enforcement (ICE) has a new budget under the current administration, and they are going on a surveillance tech shopping spree. Standing at $28.7 billion dollars for the year 2025 (nearly triple their 2024 budget) and at least another $56.25 billion over the next three years, ICE's budget would be the envy of many national militaries around the world. Indeed, this budget would put ICE as the 14th most well-funded military in the world, right between Ukraine and Israel.
There are many different agencies under U.S. Department of Homeland Security (DHS) that deal with immigration, as well as non-immigration related agencies such as Cybersecurity and Infrastructure Security Agency (CISA) and Federal Emergency Management Agency (FEMA). ICE is specifically the enforcement arm of the U.S. immigration apparatus. Their stated mission is to “[p]rotect America through criminal investigations and enforcing immigration laws to preserve national security and public safety.”
Of course, ICE doesn’t just end up targeting, surveilling, harassing, assaulting, detaining, and torturing people who are undocumented immigrants. They have targeted people on work permits, asylum seekers, permanent residents (people holding “green cards”), naturalized citizens, and even citizens by birth.
While the NSA and FBI might be the first agencies that come to mind when thinking about surveillance in the U.S., ICE should not be discounted. ICE has always engaged in surveillance and intelligence-gathering as part of their mission. A 2022 report by Georgetown Law’s Center for Privacy and Technology found the following:
With a budget for 2025 that is 10 times the size of the agency’s total surveillance spending over the last 13 years, ICE is going on a shopping spree, creating one of the largest, most comprehensive domestic surveillance machines in history.
The entire surveillance industry has been allowed to grow and flourish under both Democratic and Republican regimes. For example, President Obama dramatically expanded ICE from its more limited origins, while at the same time narrowing its focus to undocumented people accused of crimes. Under the first and second Trump administrations, ICE ramped up its operations significantly, increasing raids in major cities far from the southern border and casting a much wider net on potential targets. ICE has most recently expanded its partnerships with sheriffs across the U.S., and deported more than 1.5 million people cumulatively under the Trump administrations (600,000 of those were just during the first year of Trump’s second term according to DHS statistics), not including the 1.6 million people DHS claims have “self-deported.” More horrifying is that in just the last year of the current administration, 4,250 people detained by ICE have gone missing, and 31 have died in custody or while being detained. In contrast, 24 people died in ICE custody during the entirety of the Biden administration.
ICE also has openly stated that they plan to spy on the American public, looking for any signs of left-wing dissent against their domestic military-like presence. Acting ICE Director Todd Lyons said in a recent interview that his agency “was dedicated to the mission of going after” Antifa and left-wing gun clubs.
On a long enough timeline, any surveillance tool you build will eventually be used by people you don’t like for reasons that you disagree with.
On a long enough timeline, any surveillance tool you build will eventually be used by people you don’t like for reasons that you disagree with. A surveillance-industrial complex and a democratic society are fundamentally incompatible, regardless of your political party.
EFF recently published a guide to using government databases to dig up homeland security spending and compiled our own dataset of companies selling tech to DHS components. In 2025, ICE entered new contracts with several private companies for location surveillance, social media surveillance, face surveillance, spyware, and phone surveillance. Let’s dig into each.
One common surveillance tactic of immigration officials is to get physical access to a person’s phone, either while the person is detained at a border crossing, or while they are under arrest. ICE renewed an $11 million contract with a company called Cellebrite, which helps ICE unlock phones and then can take a complete image of all the data on the phone, including apps, location history, photos, notes, call records, text messages, and even Signal and WhatsApp messages. ICE also signed a $3 million contract with Cellebrite’s main competitor Magnet Forensics, makers of the Graykey device for unlocking phones. DHS has had contracts with Cellebrite since 2008, but the number of phones they search has risen dramatically each year, reaching a new high of 14,899 devices searched by ICE’s sister agency U.S. Customs and Border Protection (CBP) between April and June of 2025.
If ICE can’t get physical access to your phone, that won’t stop them from trying to gain access to your data. They have also resumed a $2 million contract with the spyware manufacturer, Paragon. Paragon makes the Graphite spyware, which made headlines in 2025 for being found on the phones of several dozen members of Italian civil society. Graphite is able to harvest messages from multiple different encrypted chat apps such as Signal and WhatsApp without the user ever knowing.
Our concern with ICE buying this software is the likelihood that it will be used against undocumented people and immigrants who are here legally, as well as U.S. citizens who have spoken up against ICE or who work with immigrant communities. Malware such as Graphite can be used to read encrypted messages as they are sent, other forms of spyware can also download files, photos, location history, record phone calls, and even discretely turn on your microphone to record you.
The most effective way to protect yourself from smartphone surveillance would be to not have a phone. But that’s not realistic advice in modern society. Fortunately, for most people there are other ways you can make it harder for ICE to spy on your digital life.
The first and easiest step is to keep your phone up to date. Installing security updates makes it harder to use malware against you and makes it less likely for Cellebrite to break into your phone. Likewise, both iPhone (Lockdown Mode) and Android (Advanced Protection) offer special modes that lock your phone down and can help protect against some malware.
The first and easiest step is to keep your phone up to date.
Having your phone’s software up to date and locked with a strong alphanumeric password will offer some protection against Cellebrite, depending on your model of phone. However, the strongest protection is simply to keep your phone turned off, which puts it in “before first unlock” mode and has been typically harder for law enforcement to bypass. This is good to do if you are at a protest and expect to be arrested, if you are crossing a border, or if you are expecting to encounter ICE. Keeping your phone on airplane mode should be enough to protect against cell-site simulators, but turning your phone off will offer extra protection against cell-site simulators and Cellebrite devices. If you aren’t able to turn your phone off, it’s a good idea to at least turn off face/fingerprint unlock to make it harder for police to force you to unlock your phone. While EFF continues to fight to strengthen our legal protections against compelling people to decrypt their devices, there is currently less protection against compelled face and fingerprint unlocking than there is against compelled password disclosure.
ICE has also spent $5 million to acquire at least two location and social media surveillance tools: Webloc and Tangles, from a company called Pen Link, an established player in the open source intelligence space. Webloc gathers the locations of millions of phones by gathering data from mobile data brokers and linking it together with other information about users. Tangles is a social media surveillance tool which combines web scraping with access to social media application programming interfaces. These tools are able to build a dossier on anyone who has a public social media account. Tangles is able to link together a person’s posting history, posts, and comments containing keywords, location history, tags, social graph, and photos with those of their friends and family. Penlink then sells this information to law enforcement, allowing law enforcement to avoid the need for a warrant. This means ICE can look up historic and current locations of many people all across the U.S. without ever having to get a warrant.
These tools are able to build a dossier on anyone who has a public social media account.
ICE also has established contracts with other social media scanning and AI analysis companies, such as a $4.2 million contract with a company called Fivecast for the social media surveillance and AI analysis tool ONYX. According to Fivecast, ONYX can conduct “automated, continuous and targeted collection of multimedia data” from all major “news streams, search engines, social media, marketplaces, the dark web, etc.” ONYX can build what it calls “digital footprints” from biographical data and curated datasets spanning numerous platforms, and “track shifts in sentiment and emotion” and identify the level of risk associated with an individual.
Another contract is with ShadowDragon for their product Social Net, which is able to monitor publicly available data from over 200 websites. In an acquisition document from 2022, ICE confirmed that ShadowDragon allowed the agency to search “100+ social networking sites,” noting that “[p]ersistent access to Facebook and Twitter provided by ShadowDragon SocialNet is of the utmost importance as they are the most prominent social media platforms.”
ICE has also indicated that they intend to spend between 20 and 50 million dollars on building and staffing a 24/7 social media monitoring office with at least 30 full time agents to comb every major social media website for leads that could generate enforcement raids.
For U.S. citizens, making your account private on social media is a good place to start. You might also consider having accounts under a pseudonym, or deleting your social media accounts altogether. For more information, check out our guide to protecting yourself on social media. Unfortunately, people immigrating to the U.S. might be subject to greater scrutiny, including mandatory social media checks, and should consult with an immigration attorney before taking any action. For people traveling to the U.S., new rules will soon likely require them to reveal five years of social media history and 10 years of past email addresses to immigration officials.
But it’s not just your digital habits ICE wants to surveil; they also want to spy on you in the physical world. ICE has contracts with multiple automated license plate reader (ALPR) companies and is able to follow the driving habits of a large percentage of Americans. ICE uses this data to track down specific people anywhere in the country. ICE has a $6 million contract through a Thomson Reuters subsidiary to access ALPR data from Motorola Solutions. ICE has also persuaded local law enforcement officers to run searches on their behalf through Flock Safety's massive network of ALPR data. CBP, including Border Patrol, also operates a network of covert ALPR systems in many areas.
ICE has also invested in biometric surveillance tools, such as face recognition software called Mobile Fortify to scan the faces of people they stop to determine if they are here legally. Mobile Fortify checks the pictures it takes against a database of 200 million photos for a match (the source of the photos is unknown). Additionally, ICE has a $10 million contract with Clearview AI for face recognition. ICE has also contracted with iris scanning company BI2 technologies for even more invasive biometric surveillance. ICE agents have also been spotted wearing Meta’s Ray-Ban video recording sunglasses.
ICE has acquired trucks equipped with cell-site simulators (AKA Stingrays) from a company called TechOps Specialty Vehicles (likely the cell-site simulators were manufactured by another company). This is not the first time ICE has bought this technology. According to documents obtained by the American Civil Liberties Union, ICE deployed cell-site simulators at least 466 times between 2017 and 2019, and ICE more than 1,885 times between 2013 and 2017, according to documents obtained by BuzzFeed News. Cell-site simulators can be used to track down a specific person in real time, with more granularity than a phone company or tools like Webloc can provide, though Webloc has the distinct advantage of being used without a warrant and not requiring agents to be in the vicinity of the person being tracked.
Taking public transit or bicycling is a great way to keep yourself off ALPR databases, but an even better way is to go to your local city council meetings and demand the city cancels contracts with ALPR companies, like people have done in Flagstaff, Arizona; Eugene, Oregon; and Denver, Colorado, among others.
If you are at a protest, putting your phone on airplane mode could help protect you from cell-site simulators and from apps on your phone disclosing your location, but might leave you vulnerable to advanced targeted attacks. For more advanced protection, turning your phone completely off protects against all radio based attacks, and also makes it harder for tools like Cellebrite to break into your phone as discussed above. But each individual will need to weigh their need for security from advanced radio based attacks against their need to document potential abuses through photo or video. For more information about protecting yourself at a protest, head over to SSD.
There is nothing you can do to change your face, which is why we need more stringent privacy laws such as Illinois’ Biometric Information Privacy Act.
Last but not least, ICE uses tools to combine and search all this data along with the data on Americans they have acquired from private companies, the IRS, TSA, and other government databases.
To search all this data, ICE uses ImmigrationOS, a system that came from a $30-million contract with Palantir. What Palantir does is hard to explain, even for people who work there, but essentially they are plumbers. Palantir makes it so that ICE has all the data they have acquired in one place so it’s easy to search through. Palantir links data from different databases, like IRS data, immigration records, and private databases, and enables ICE to view all of this data about a specific person in one place.
Palantir makes it so that ICE has all the data they have acquired in one place so it’s easy to search through.
The true civil liberties nightmare of Palantir is that they enable governments to link data that should have never been linked. There are good civil liberties reasons why IRS data was never linked with immigration data and was never linked with social media data, but Palantir breaks those firewalls. Palantir has labeled themselves as a progressive, human rights centric company historically, but their recent actions have given them away as just another tech company enabling surveillance nightmares.
Understanding the capabilities and limits of ICE and how to threat model helps you and your community fight back, remain powerful, and protect yourself.
One of the most important things you can do is to not spread rumors and misinformation. Rumors like “ICE has malware so now everyone's phones are compromised” or “Palantir knows what you are doing all the time” or “Signal is broken” don’t help your community. It’s more useful to spread facts, ways to protect yourself, and ways to fight back. For information about how to create a security plan for yourself or your community, and other tips to protect yourself, read our Surveillance Self-Defense guides.
One way to fight back against ICE is in the courts. EFF currently has a lawsuit against ICE over their pressure on Apple and Google to take down ICE spotting apps, like ICEBlock. We also represent multiple labor unions suing ICE over their social media surveillance practices.
We have also demanded the San Francisco Police Department stop sharing data illegally with ICE, and issued a statement condemning the collaboration between ICE and the malware provider Paragon. We also continue to maintain our Rayhunter project for detecting cell-site simulators.
Other civil liberties organizations are also suing ICE. ACLU has sued ICE over a subpoena to Meta attempting to identify the owner of an account providing advice to protestors, and another coalition of groups has thus far successfully sued the IRS to stop sharing taxpayer data with ICE.
We need to have a hard look at the surveillance industry. It is a key enabler of vast and untold violations of human rights and civil liberties, and it continues to be used by aspiring autocrats to threaten our very democracy. As long as it exists, the surveillance industry, and the data it generates, will be an irresistible tool for anti-democratic forces.
Age verification mandates are spreading fast, and they’re ushering in a new age of online surveillance, censorship, and exclusion for everyone—not just young people. Age-gating laws generally require websites and apps to collect sensitive data from every user, often through invasive tools like ID checks, biometric scans, or other dubious “estimation” methods, before granting them access to certain content or services. Lawmakers tout these laws as the silver-bullet solution to “kids’ online safety,” but in reality, age-verification mandates wall off large swaths of the web, build sweeping new surveillance infrastructure, increase the risk of data breaches and real-life privacy harms, and threaten the anonymity that has long allowed people to seek support, explore new ideas, and organize and build community online.
Join EFF's Rindala Alajaji and Alexis Hancock along with Hana Memon from Gen-Z for Change and Cynthia Conti-Cook from Collaborative Research Center for Resilience for a conversation about what we stand to lose as more and more governments push to age-gate the web. We’ll break down how these laws work, who they exclude, and how these mandates threaten privacy and free expression for people of all ages. The conversation will be followed by a live Q&A.
This event will be live-captioned and recorded. EFF is committed to improving accessibility for our events. If you have any accessibility questions regarding the event, please contact events@eff.org.
EFF is dedicated to a harassment-free experience for everyone, and all participants are encouraged to view our full Event Expectations.
Want to make sure you don’t miss our next livestream? Here’s a link to sign up for updates about this series: eff.org/ECUpdates. If you have a friend or colleague that might be interested, please join the fight for your digital rights by forwarding this link: eff.org/EFFectingChange. Thank you for helping EFF spread the word about privacy and free expression online.
We hope you and your friends can join us live! If you can't make it, we’ll post the recording afterward on YouTube and the Internet Archive!
This guide was co-written by Andrew Zuker with support from the Heinrich Boell Foundation.
The U.S. government publishes volumes of detailed data on the money it spends, but searching through it and finding information can be challenging. Complex search functions and poor user interfaces on government reporting sites can hamper an investigation, as can inconsistent company profiles and complex corporate ownership structures.
This week, EFF and the Heinrich Boell Foundation released an update to our database of vendors providing technology to components of the U.S. Department of Homeland Security (DHS), such as Immigration and Customs Enforcement (ICE) and Customs and Border Protections (CBP). It includes new vendor profiles, new fields, and updated data on top contractors, so that journalists and researchers have a jumping-off point for their own investigations.
Access the dataset through Google Sheets (Google's Privacy Policy applies) or download the Excel file here.
This time we thought we would also share some of the research methods we developed while assembling this dataset.
This guide covers the key databases that store information on federal spending and contracts (often referred to as "awards"), government solicitations for products and services, and the government's "online shopping superstore," plus a few other deep-in-the-weeds datasets buried in the online bureaucracy. We have provided a step-by-step guide for searching these sites efficiently and help tips for finding information. While we have written this specifically with DHS agencies in mind, it should serve as a useful resource for procurement across the federal government.
The Federal Procurement Data System (FPDS) is the best place to start for finding out what companies are working with DHS. It is the official system for tracking federal discretionary spending and contains current data on contracts with non-governmental entities like corporations and private businesses. Award data is up-to-date and includes detailed information on vendors and awards which can be helpful when searching the other systems. It is a little bit old-school, but that often makes it one of the easiest and quickest sites to search, once you get the hang of it, since it offers a lot of options for narrowing search parameters to specific agencies, vendors, classification of services, etc.
How to Use FDPS
To begin searching Awards for a particular vendor, click into the “ezSearch” field in the center of the page, delete or replace the text “Google-like search to help you find federal contracts…” with a vendor name or keywords, and hit Enter to begin a new search.
A new tab will open automatically with exact matches at the top.
Four “Top 10” modules on the left side of the page link to top results in descending order: Department Full Name, Contracting Agency Name, Full Legal Business Name, and Treasury Account Symbol. These ranked lists help the user quickly narrow in on departments and agencies that vendors do business with. DHS may not appear in the “Top 10” results, which may indicate that the vendor hasn’t yet been awarded DHS or subagency contracts.
For example, if you searched the term “FLIR”, as in Teledyne FLIR who make infrared surveillance systems used along the U.S.-Mexico border, DHS is the 2nd result in the “Top 10: Department Full Name” box.
To see all DHS contracts awarded to the vendor, click “Homeland Security, Department of” from the “Top 10 Department Full Name” module. When the page loads, you will see the subcomponents of DHS (e.g., ICE, CBP, or the U.S. Secret Service) in the lefthand menu. You can click on each of those to drill down even further. You can also drill down by choosing a company.
Sorting options can be found on the right side of the page which offer the ability to refine and organize search results. One of the most useful is "Date Signed," which will arrange the results in chronological order.
You don't have to search by a company name. You can also use a product keyword, such as "LPR" (license plate reader). However, because keywords are not consistently used by government agencies, you will need to try various permutations to gather the most data.
Each click or search filter adds a new term to the search both in the main field at the top and in the Search Criteria module on the right. They can be deleted by clicking the X next to the term in this module or by removing the text in the main search field.
For each contract item, you can click "View" to see the specific details. However, these pages don't have permalinks, so you'll want to print-to-pdf if you need to retain a permanent copy of the record.
Often the vendor brand name we know from their marketing or news media is not the same entity that is awarded government contracts. Foreign companies in particular rely on partnerships with domestic entities that are established federal contractors. If you can’t find any spending records for a vendor, search the web for information on the company including acquisitions, partnerships, licensing agreements, parent companies, and subsidiaries. It is likely that one of these types of related companies is the contract holder.
The Federal Funding and Accountability Act (FFATA) of 2006 and the DATA Act of 2014 require the government to publish all spending records and contracts on a single, searchable public website, including agency-specific contracts, using unified reporting standards to ensure consistent, reliable, searchable data. This led to the creation of USA Spending (usaspending.gov).
USA Spending is populated with data from multiple sources including the Federal Procurement Data System (fpds.gov) and the System for Awards Management (sam.gov - which we'll discuss in the next section). It also compiles Treasury Reports and data from the financial systems of dozens of federal agencies. We relied heavily on Awards data from these systems to verify vendor information including contracts with the DHS and its subagencies such as CBP and ICE.
USA Spending has a more modern interface, but is often very slow with the information often hidden in expandable menus. In many ways it is duplicative of FPDS, but with more features, including the ability to bookmark individual pages. We often found ourselves using FPDS to quickly identify data, and then using the "Award ID" number to find the specific record within USA Spending.
USA Spending also has some visualizations and ways to analyze data in chart form, which is not possible with the largely text-based FPDS.
How to Use USA Spending
To begin searching for DHS awards, click on either “Search Award Data” on the navigation bar, or the blue “Start Searching Awards”button.
On the left of the Search page are a list of drop down menus with options. You can enter a vendor name as a keyword, or expand the “Recipient” menu if you know the full company name or their Unique Entity Identifier (UEI) number. Expand the “Agency Tab” and enter DHS which will bring up the Department of Homeland Security Option.
In the example below, we entered “Palantir Technologies” as a keyword, and selected DHS in the Agency dropdown:
For vendors with hundreds of contracts that return many pages of results, consider adding more filters to the search such as a specific time period or specifying a Funding Agency such as ICE or CBP. In this example, the filters “Palantir Technologies” and “DHS” returned 13 results (at the time of publication). It is important to note that the search results table is larger than what displays in that module. You can scroll down to view more Awards and scroll to the right to see much more information.
Scroll down outside of that module to reveal more info including modules for Results by Category, Results over Time, and Results by Geography, all of which can be viewed as a list or graph.
Once you've identified a contract, you can click the "Prime Award ID" to see the granular details for each time.
From the search, you can also select just the agency to see all the contracts on file. Each agency also has its own page showing a breakdown for every fiscal year of how much money they had to spend and which components spent the most. For example, here's DHS's page.
So far we've talked about how to track contracts and spending, but now let's take a step back and look at how those contracts come to be. The System for Award Management, SAM.gov, is the site that allows companies to see what products and services the government intends to buy so they can bid on the contract. But SAM.gov is also open to the public, which means you can see the same information, including a detailed scope of a project and sometimes even technical details.
How to Use Sam.gov
SAM.gov does not require an account for its basic contracting opportunity searches, but you may want to create one in order to save the things you find and to receive keyword- or agency-based alerts via email when new items of interest are posted.
First you will click "Search" in the menu bar, which will bring you to this page:
We recommend selecting both "Active" and "Inactive" in the Status menu. Contracts quickly go inactive, and besides, sometimes the contracts you are most interested in are several years old.
If you are researching a particular technology such as unmanned aerial vehicles, you might just type "unmanned" in the Simple Search bar. That will bring up every solicitation with that keyword across the federal government.
One of the most useful features is filtering by agency, while leaving the keyword search blank. This will return a running list of an agency's calls for bids and related procurement activities. It is worth checking regularly. For example, here's what CBP's looks like on a given day:
If you click on an item, you should next scroll down to see if there are attachments. These tend to contain the most details. Specifically, you should look for the term "SOW," the abbreviation for "Statement of Work." For example, here are the attachments for a CBP contracting opportunity for "Cellular Covert Cameras":
The first document is the Statement of Work, which tells you the exact brand, model, and number of devices they want to acquire:
The attachments also included a "BNO Justification." BNO stands for "Brand Name Only," and this document explains in even more detail why CBP wants that specific product:
If you see the terms "Sole Source" in a listing, that also means that an agency has decided that only one product meets its requirements and it will not open bidding to other companies.
In addition to contracting, many agencies announce "Industry Day" events, usually virtual, that members of the public can join. This is a unique opportunity to listen in on what contractors are being told by government purchasing officials. The presentation slides are also often later uploaded to the SAM.gov page. Occasionally, the list of attendees will also be posted, and you'll find several examples of those lists in our dataset.
Another way to investigate DHS purchasing is by browsing the catalog of items and services immediately available to them. The General Services Administration operates GSA Advantage, which it describes as "the government's central online shopping superstore." The website's search is open, allowing members of the public to view any vendors' offerings–including both products and services– easily as they would with any online marketplace.
For example, you could search for "license plate reader" and produce a list of available products:
If you click "Advanced Search," you can also isolate every product available from a particular manufacturer. For example, here are the results when you search for products available from Skydio, a drone manufacturer.
If you switch from "Products" to "Services" you can export datasets for each company about their offerings. For example, if you search for "Palantir" you'll get results that look like this:
This means all these companies are offering some sort of Palantir-related services. If you click "Matches found in Terms and Conditions," you'll download a PDF with a lot of details about what the company offers.
For example, here's a a screengrab from Anduril's documentation:
If you click "Matches Found in Price List" you'll download a spreadsheet that serves as a blueprint of what the company offers, including contract personnel. Here's a snippet from Palantir's:
Daily Public Report of Covered Contract Awards - Maybe FPDS isn't enough for you and you want to know every day what contracts have been signed. Buried in the DHS website are links to a daily feed of all contracts worth $4 million or more. It's available in XML, JSON, CSV and XLSX formats.
DHS Acquisition Planning Forecast System (APFS) - DHS operates a site for vendors to learn about upcoming contracts greater than $350,000. You can sort by agency at a granular level, such as upcoming projects by ICE Enforcement & Removal Operations. This is one to check regularly for updates.
DHS Artificial Intelligence Use Case Inventory - Many federal agencies are required to maintain datasets of "AI Use Cases." DHS has broken these out for each of its subcomponents, including ICE and CBP. Advanced users will find the spreadsheet versions of these inventory more interesting.
NASA Solutions for Enterprise-Wide Procurement (SEWP) - SEWP is a way for agencies to fast track acquisition of "Information Technology, Communication and Audio Visual" products through existing contracts. The site provides an index of existing contract holders, but the somewhat buried "Provider Lookup" has a more comprehensive list of companies involved in this type of contracting, illustrating how the companies serve as passthroughs for one another. Relatedly, DHS's list of "Prime Contractors" shows which companies hold master contracts with the agency and its components.
TechInquiry - Techinquiry is a small non-profit that aggregates records from a wide variety of sources about tech companies, particularly those involved in government contracting.
2025 was the year age verification went from a fringe policy experiment to a sweeping reality across the United States. Half of the U.S. now mandates age verification for accessing adult content or social media platforms. Nine states saw their laws take effect this year alone, with more coming in 2026.
The good news is that courts have blocked many of the laws seeking to impose age-verification gates on social media, largely for the same reasons that EFF opposes these efforts. Age-verification measures censor the internet and burden access to online speech. Though age-verification mandates are often touted as "online safety" measures for young people, the laws actually do more harm than good. They undermine the fundamental speech rights of adults and young people alike, create new barriers to internet access, and put at risk all internet users' privacy, anonymity, and security.
If you're feeling overwhelmed by this onslaught of laws and the invasive technologies behind them, you're not alone. That's why we've launched EFF's Age Verification Resource Hub at EFF.org/Age—a one-stop shop to understand what these laws actually do, what's at stake, why EFF opposes all forms of age verification, how to protect yourself, and how to join the fight for a free, open, private, and safe internet. Moreover, there is hope. Although the Supreme Court ruled that imposing age-verification gates to access adult content does not violate the First Amendment on its face, the legal fight continues regarding whether those laws are constitutional.
As we built the hub throughout 2025, we also fought state mandates in legislatures, courts, and regulatory hearings. Here's a summary of what happened this year.
Nine states’ age verification laws for accessing adult content went into effect in 2025:
Predictably, users didn’t stop accessing adult content after the laws went into effect, they just changed how they got to it. As we’ve said elsewhere: the internet always routes around censorship.
In fact, research from the New York Center for Social Media and Politics and the public policy nonprofit the Phoenix Center confirm what we’ve warned from the beginning: age verification laws don’t work. Their research found:
As foretold, when platforms block access or require invasive verification, it drives people to sites that operate outside the law—platforms that often pose greater safety risks. Instead of protecting young people, these laws push them toward less secure, less regulated spaces.
Earlier this year, we raised the alarm that state legislatures wouldn’t stop at adult content. Sure enough, throughout 2025, lawmakers set their sights on young people’s social media usage, passing laws that require platforms to verify users’ ages and obtain parental consent for accounts belonging to anyone under 18. Four states already passed similar laws in previous years. These laws were swiftly blocked in courts because they violate the First Amendment and subject every user to surveillance as a condition of participation in online speech.
And it doesn’t stop with age verification. California and Minnesota passed new laws this year requiring social media platforms to display warning labels to users. Virginia’s SB 854, which also passed this year, took a different approach. It requires social media platforms to use “commercially reasonable efforts” to determine a user's age and, if that user is under 16, limits them to one hour per day per application by default unless a parent changes the time allowance.
EFF is opposed to these laws as they have serious First Amendment concerns. And courts have agreed: in November 2025, the U.S. District Court for the District of Colorado temporarily halted Colorado's warning label law, which would have required platforms to display warnings to users under 18 about the negative impacts of social media. We expect courts to similarly halt California and Minnesota’s laws.
2025 also saw the rise of device-level and app-store age verification laws, which shift the obligation to verify users onto app stores and operating system providers. These laws seriously impact users’ (adults and young people alike) from accessing information, particularly since these laws block a much broader swath of content (not only adult or sexual content), but every bit of content provided by every application. In October, California Governor Gavin Newsom signed the Digital Age Assurance Act (AB 1043), which takes a slightly different approach to age verification in that it requires “operating system providers”—not just app stores—to offer an interface at device/account setup that prompts the account holder to indicate the user’s birth date or age. Developers must request an age signal when applications are downloaded and launched. These laws expand beyond earlier legislation passed in other states that mandate individual websites implement the law, and apply the responsibility to app stores, operating systems, or device makers at a more fundamental level.
Again, these laws have drawn legal challenges. In October, the Computer & Communications Industry Association (CCIA) filed a lawsuit arguing that Texas’s SB 2420 is unconstitutional. A separate suit, Students Engaged in Advancing Texas (SEAT) v. Paxton, challenges the same law on First Amendment grounds, arguing it violates the free speech rights of young people and adults alike. Both lawsuits argue that the burdens placed on platforms, developers, and users outweigh any proposed benefits.
States with existing laws have also begun the process of rulemaking—translating broad statutory language into specific regulatory requirements. These rulemaking processes matter, because the specific technical requirements, data—handling procedures, and enforcement mechanisms will determine just how invasive these laws become in practice.
California’s Attorney General held a hearing in November to solicit public comment on methods and standards for age assurance under SB 976, the “Protecting Our Kids from Social Media Addiction Act,” which will require age verification by the end of 2026. EFF supported the legal challenge to S.B. 976 since its passage, and federal courts have blocked portions of the law from taking effect. Now in the rulemaking process, EFF submitted comments raising concerns about the discriminatory impacts of any proposed regulations.
New York's Attorney General also released proposed rules for the state’s Stop Addictive Feeds Exploitation (SAFE) for Kids Act, describing which companies must comply and the standards for determining users’ age and obtaining parental consent. EFF submitted comments opposing the age verification requirements in September of 2024, and again in December 2025.
Our comments in both states warn that these rules risk entrenching invasive age verification systems and normalizing surveillance as a prerequisite for online participation.
As we’ve said, age verification will not stop at adult content and social media. Lawmakers are already proposing bills to require ID checks for everything from skincare products in California to diet supplements in Washington. Lawmakers in Wisconsin and Michigan have set their targets on virtual private networks, or VPNs—proposing various legislation that would ban the use of VPNs to prevent people from bypassing age verification laws. AI chatbots are next on the list, with several states considering legislation that would require age verification for all users. Behind the reasonable-sounding talking points lies a sprawling surveillance regime that would reshape how people of all ages use the internet. EFF remains ready to push back against these efforts in legislatures, regulatory hearings, and court rooms.
2025 showed us that age verification mandates are spreading rapidly, despite clear evidence that they don't work and actively harm the people they claim to protect. 2026 will be the year we push back harder—like the future of a free, open, private, and safe internet depends on it.
This is why we must fight back to protect the internet that we know and love. If you want to learn more about these bills, visit EFF.org/Age
This article is part of our Year in Review series. Read other articles about the fight for digital rights in 2025.
Our Surveillance Self-Defense (SSD) guides, which provide practical advice and explainers for how to deal with government and corporate surveillance, had a big year. We published several large updates to existing guides and released three all new guides. And with frequent massive protests across the U.S., our guide to attending a protest remained one of the most popular guides of the year, so we made sure our translations were up to date.
We started this year by taking a deep look at our various encryption guides, which start with the basics before moving up to deeper concepts. We slimmed each guide down and tried to focus on making them as clear and concise as deep explainers on complicated topics can be. We reviewed and edited four guides in total:
And if you’re not sure where to start, we got you covered with the new Interested in Encryption? playlist.
We launched three new guides this year, including iPhone and Android privacy guides, which walk you through all the various privacy options of your phone. Both of these guides received a handful of updates throughout their first year as new features were released or, in the case of the iPhone, a new design language was introduced. These also got a fun little boost from a segment on "Last Week Tonight with John Oliver" telling people how to disable their phone’s advertising identifier.
We also launched our How to: Manage Your Digital Footprint guide. This guide is designed to help you claw back some of the data you may find about yourself online, walking through different privacy options across different platforms, digging up old accounts, removing yourself from people search sites, and much more.
As is the case with most software, there is always incremental work to do. This year, that meant small updates to our WhatsApp and Signal guides to acknowledge new features (both are already on deck for similar updates early next year as well).
We overhauled our device encryption guides for Windows, Mac, and Linux, rolling what was once three guides into one, and including more detailed guidance on how to handle recovery keys. Some slight changes to how this works on both Windows and Mac means this one will get another look early next year as well.
Speaking of rolling multiple guides into one, we did the same with our guidance for the Tor browser, where it once lived across three guides, it now lives as one that covers all the major desktop platforms (the mobile guide remains separate).
The password manager guide saw some small changes to note some new features with Apple and Chrome’s managers, as well as some new independent security audits. Likewise, the VPN guide got a light touch to address the TunnelVision security issue.
Finally, the secure deletion guide got a much needed update after years of dormancy. With the proliferation of solid state drives (SSDs, not to be confused with SSD), not much has changed in the secure deletion space, but we did move our guidance for those SSDs to the top of the guide to make it easier to find, while still acknowledging many people around the world still only have access to a computer with spinning disk drives.
As always, we worked on translations for these updates. We’re very close to a point where every current SSD guide is updated and translated into Arabic, French, Mandarin, Portuguese, Russian, Spanish, and Turkish.
And with the help of Localization Lab, we also now have translations for a handful of the most important guides in Changana, Mozambican Portuguese, Ndau, Luganda, and Bengali.
Sometimes we take our SSD-like advice and blog it so we can respond to news events or talk about more niche topics. This year, we blogged about new features, like WhatsApp’s “Advanced Chat Privacy” and Google’s "Advanced Protection.” We also broke down the differences between how different secure chat clients handle backups and pushed for expanding encryption on Android and iPhone.
We fight for more privacy and security every day of every year, but until we get that, stronger controls of our data and a better understanding of how technology works is our best defense.
This article is part of our Year in Review series. Read other articles about the fight for digital rights in 2025.
In the name of 'protecting kids online,' Congress pushed forward legislation this year that could have severely undermined our privacy and stifled free speech. These bills would have mandated invasive age-verification checks for everyone online—adults and kids alike—handing unprecedented control to tech companies and government authorities.
Lawmakers from both sides of the aisle introduced bill after bill, each one somehow more problematic than the last, and each one a gateway for massive surveillance, internet censorship, and government overreach. In all, Congress considered nearly twenty federal proposals.
For us, this meant a year of playing legislative whack-a-mole, fighting off one bad bill after another. But more importantly, it meant building sustained opposition, strengthening coalitions, and empowering our supporters—that's you!—with the tools you need to understand what's at stake and take action.
Luckily, thanks to this strong opposition, these federal efforts all stalled… for now.
So, before we hang our hats and prepare for the new year, let’s review some of our major wins against federal age-verification legislation in 2025.
Of the dozens of federal proposals relating to kids online, the Kids Online Safety Act remains the biggest threat. We, along with a coalition of civil liberties groups, LGBTQ+ advocates, youth organizations, human rights advocates, and privacy experts, have been sounding the alarm on KOSA for years now.
First introduced in 2022, KOSA would allow the Federal Trade Commission to sue apps and websites that don’t take measures to restrict young people’s access to certain content. There have been numerous versions introduced, though all of them share a common core: KOSA is an unconstitutional censorship bill that threatens the speech and privacy rights of all internet users. It would impose a requirement that platforms “exercise reasonable care” to prevent and mitigate a sweeping list of harms to minors, including depression, anxiety, eating disorders, substance use, bullying, and “compulsive usage.” Those prohibitions are so broad that they will sweep up online speech about the topics, including efforts to provide resources to adults and minors experiencing them. The bill claims prohibit censorship based on “the viewpoint of users,” but that’s simply a smokescreen. Its core function is to let the federal government sue platforms, big or small, that don’t block or restrict content that someone later claims contributed to one of these harms.
In addition to stifling online speech, KOSA would strongly incentivize age-verification systems—forcing all users, adults and minors, to prove who they are before they can speak or read online. Because KOSA requires online services to separate and censor aspects of their services accessed by children, services are highly likely to demand to know every user’s age to avoid showing minors any of the content KOSA deems harmful. There are a variety of age determination options, but all have serious privacy, accuracy, or security problems. Even worse, age-verification schemes lead everyone to provide even more personal data to the very online services that have invaded our privacy before. And all age verification systems, at their core, burden the rights of adults to read, get information, and speak and browse online anonymously.
Despite what lawmakers claim, KOSA won’t bother big tech—in fact, they endorse it! The bill is written so that big tech companies, like Apple and X, will be able to handle the regulatory burden that KOSA will demand, while smaller platforms will struggle to comply. Under KOSA, a small platform hosting mental health discussion boards will be just as vulnerable as Meta or TikTok—but much less able to defend itself.
The good news is that KOSA’s momentum this Congress was waning at best. There was a lot of talk about the bill from lawmakers, but little action. The Senate version of the bill, which passed overwhelmingly last summer, did not even make it out of committee this Congress.
In the House, lawmakers could not get on the same page about the bill—so much so that one of the original sponsors of KOSA actually voted against the bill in committee in December.
The bad news is that lawmakers are determined to keep raising this issue, as soon as the beginning of next year. So let’s keep the momentum going by showing them that users do not want age verification mandates—we want privacy.
Don't let congress censor the internet
KOSA wasn’t the only federal bill in 2025 that used “kids’ safety” as a cover for sweeping surveillance and censorship mandates. Concern about possible harms of AI chatbots dominated policy discussion this year in Congress.
One of the most alarming proposals on the issue was the GUARD Act, which would require AI chatbots to verify all users’ ages, prohibit minors from using AI tools, and implement steep criminal penalties for chatbots that promote or solicit certain harms. As we wrote in November, though the GUARD Act may look like a child-safety bill, in practice it’s an age-gating mandate that could be imposed on nearly every public-facing AI chatbot—from customer-service bots to search-engine assistants. The GUARD Act could force countless AI companies to collect sensitive identity data, chill online speech, and block teens from using some of the digital tools that they rely on every day.
Like KOSA, the GUARD Act would make the internet less free, less private, and less safe for everyone. It would further consolidate power and resources in the hands of the bigger AI companies, crush smaller developers, and chill innovation under the threat of massive fines. And it would cut off vulnerable groups’ ability to use helpful everyday AI tools, further fracturing the internet we know and love.
With your help, we urged lawmakers to reject the GUARD Act and focus instead on policies that provide more transparency, options, and comprehensive privacy for all users.
Together, these bills reveal a troubling pattern in Congress this year. Rather than actually protecting young people’s privacy and safety online, Congress continues to push a legislative framework that’s based on some deeply flawed assumptions:
We’ve written over and over about the many communities who are immeasurably harmed by online age verification mandates. It is also worth remembering who these bills serve—big tech companies, private age verification vendors, AI companies, and legislators vying for the credit of “solving” online safety while undermining users at every turn.
We fought these bills all through 2025, and we’ll continue to do so until we beat age verification for good. So rest up, read up (starting with our all-new resource hub, EFF.org/Age!), and get ready to join us in this fight in 2026. Thank you for your support this year.
This article is part of our Year in Review series. Read other articles about the fight for digital rights in 2025.
Lawmakers in at least a dozen states believe that they can pass laws blocking young people from social media or require them to get their parents’ permission before logging on. Fortunately, nearly every trial court to review these laws has ruled that they are unconstitutional.
It’s not just courts telling these lawmakers they are wrong. EFF has spent the past year filing friend-of-the-court briefs in courts across the country explaining how these laws violate young people’s First Amendment rights to speak and get information online. In the process, these laws also burden adults’ rights, and jeopardize everyone’s privacy and data security.
Minors have long had the same First Amendment rights as adults: to talk about politics, create art, comment on the news, discuss or practice religion, and more. The internet simply amplified their ability to speak, organize, and find community.
Although these state laws vary in scope, most have two core features. First, they require social media services to estimate or verify the ages of all users. Second, they either ban minor access to social media, or require parental permission.
In 2025, EFF filed briefs challenging age-gating laws in California (twice), Florida, Georgia, Mississippi, Ohio, Utah, Texas, and Tennessee. Across these cases we argued the same point: these laws burden the First Amendment rights of both young people and adults. In many of these briefs, the ACLU, Center for Democracy & Technology, Freedom to Read Foundation, LGBT Technology Institute, TechFreedom, and Woodhull Freedom Foundation joined.
There is no “kid exception” to the First Amendment. The Supreme Court has repeatedly struck down laws that restrict minors’ speech or impose parental-permission requirements. Banning young people entirely from social media is an extreme measure that doesn’t match the actual risks. As EFF has urged, lawmakers should pursue strong privacy laws, not censorship, to address online harms.
These laws also burden everyone’s speech requiring users to prove their age. ID-based systems of access can lock people out if they don’t have the right form of ID, and biometric systems are often discriminatory or inaccurate. Requiring users to identify themselves before speaking also chills anonymous speech—protected by the First Amendment, and essential for those who risk retaliation.
Finally, requiring users to provide sensitive personal information increases their risk of future privacy and security invasions. Most of these laws perversely require social media companies to collect even more personal information from everyone, especially children, who can be more vulnerable to identify theft.
EFF will continue to fight for the rights of minors and adults to access the internet, speak freely, and organize online.
This article is part of our Year in Review series. Read other articles about the fight for digital rights in 2025.
This year, we fought back against the return of a terrible idea that hasn’t improved with age: site blocking laws.
More than a decade ago, Congress tried to pass SOPA and PIPA—two sweeping bills that would have allowed the government and copyright holders to quickly shut down entire websites based on allegations of piracy. The backlash was massive. Internet users, free speech advocates, and tech companies flooded lawmakers with protests, culminating in an “Internet Blackout” on January 18, 2012. Turns out, Americans don’t like government-run internet blacklists. The bills were ultimately shelved.
But we’ve never believed they were gone for good. The major media and entertainment companies that backed site blocking in the US in 2012 turned to pushing for site-blocking laws in other countries. Rightsholders continued to ask US courts for site-blocking orders, often winning them without a new law. And sure enough, the Motion Picture Association (MPA) and its allies have asked Congress to try again.
There were no less than three Congressional drafts of site-blocking legislation. Representative Zoe Lofgren kicked off the year with the Foreign Anti-Digital Piracy Act (FADPA). Fellow House of Representatives member Darrell Issa also claimed to be working on a bill that would make it offensively easy for a studio to block your access to a website based solely on the belief that there is infringement happening. Not to be left out, the Senate Judiciary Committee produced the terribly named Block BEARD Act.
None of these three attempts to fundamentally alter the way you experience the internet moved too far after their press releases. But the number tells us that there is, once again, an appetite among major media conglomerates and politicians to resurrect SOPA/PIPA from the dead.
None of these proposals fixes the flaws of SOPA/PIPA, and none ever could. Site blocking is a flawed idea and a disaster for free expression that no amount of rewriting will fix. There is no way to create a fast lane for removing your access to a website that is not a major threat to the open web. Just as we opposed SOPA/PIPA over ten years ago, we oppose these efforts.
Site blocking bills seek to build a new infrastructure of censorship into the heart of the internet. They would enable court orders directed to the organizations that make the internet work, like internet service providers, domain name resolvers, and reverse proxy services, compelling them to help block US internet users from visiting websites accused of copyright infringement. The technical means haven’t changed much since 2012. - tThey involve blocking Internet Protocol addresses or domain names of websites. These methods are blunt—sledgehammers rather than scalpels. Today, many websites are hosted on cloud infrastructure or use shared IP addresses. Blocking one target can mean blocking thousands of unrelated sites. That kind of digital collateral damage has already happened in Austria, Italy, South Korea, France, and in the US, to name just a few.
Given this downside, one would think the benefits of copyright enforcement from these bills ought to be significant. But site blocking is trivially easy to evade. Determined site owners can create the same content on a new domain within hours. Users who want to see blocked content can fire up a VPN or change a single DNS setting to get back online.
The limits that lawmakers have proposed to put on these laws are an illusion. While ostensibly aimed at “foreign” websites, they sweep in any website that doesn’t conspicuously display a US origin, putting anonymity at risk. And despite the rhetoric of MPA and others that new laws would be used only by responsible companies against the largest criminal syndicates, laws don’t work that way. Massive new censorship powers invite abuse by opportunists large and small, and the costs to the economy, security, and free expression are widely borne.
It’s time for Big Media and its friends in Congress to drop this flawed idea. But as long as they keep bringing it up, we’ll keep on rallying internet users of all stripes to fight it.
This article is part of our Year in Review series. Read other articles about the fight for digital rights in 2025.
Throughout 2025, EFF conducted groundbreaking investigations into Flock Safety's automated license plate reader (ALPR) network, revealing a system designed to enable mass surveillance and susceptible to grave abuses. Our research sparked state and federal investigations, drove landmark litigation, and exposed dangerous expansion into always-listening voice detection technology. We documented how Flock's surveillance infrastructure allowed law enforcement to track protesters exercising their First Amendment rights, target Romani people with discriminatory searches, and surveil women seeking reproductive healthcare.
When we obtained datasets representing more than 12 million searches logged by more than 3,900 agencies between December 2024 and October 2025, the patterns were unmistakable. Agencies logged hundreds of searches related to political demonstrations—the 50501 protests in February, Hands Off protests in April, and No Kings protests in June and October. Nineteen agencies conducted dozens of searches specifically tied to No Kings protests alone. Sometimes searches explicitly referenced protest activity; other times, agencies used vague terminology to obscure surveillance of constitutionally protected speech.
The surveillance extended beyond mass demonstrations. Three agencies used Flock's system to target activists from Direct Action Everywhere, an animal-rights organization using civil disobedience to expose factory farm conditions. Delaware State Police queried the Flock network nine times in March 2025 related to Direct Action Everywhere actions—showing how ALPR surveillance targets groups engaged in activism challenging powerful industries.
Our November analysis revealed deeply troubling patterns: more than 80 law enforcement agencies used language perpetuating harmful stereotypes against Romani people when searching the nationwide Flock Safety ALPR network. Between June 2024 and October 2025, police performed hundreds of searches using terms such as "roma" and racial slurs—often without mentioning any suspected crime.
Audit logs revealed searches including "roma traveler," "possible g*psy," and "g*psy ruse." Grand Prairie Police Department in Texas searched for the slur six times while using Flock's "Convoy" feature, which identifies vehicles traveling together—essentially targeting an entire traveling community without specifying any crime. According to a 2020 Harvard University survey, four out of 10 Romani Americans reported being subjected to racial profiling by police. Flock's system makes such discrimination faster and easier to execute at scale.
In October, we obtained documents showing that Texas deputies queried Flock Safety's surveillance data in what police characterized as a missing person investigation, but was actually an abortion case. Deputies initiated a "death investigation" of a "non-viable fetus," logged evidence of a woman's self-managed abortion, and consulted prosecutors about possible charges.
A Johnson County official ran two searches with the note "had an abortion, search for female." The second search probed 6,809 networks, accessing 83,345 cameras across nearly the entire country. This case revealed Flock's fundamental danger: a single query accesses more than 83,000 cameras spanning almost the entire nation, with minimal oversight and maximum potential for abuse—particularly when weaponized against people seeking reproductive healthcare.
In June, EFF explained why Flock Safety's announced feature updates cannot make ALPRs safe. The company promised privacy-enhancing features like geofencing and retention limits in response to public pressure. But these tweaks don't address the core problem: Flock's business model depends on building a nationwide, interconnected surveillance network that creates risks no software update can eliminate. Our 2025 investigations proved that abuses stem from the architecture itself, not just how individual agencies use the technology.
EFF's work sparked significant accountability measures. U.S. Rep. Raja Krishnamoorthi and Rep. Robert Garcia launched a formal investigation into Flock's role in "enabling invasive surveillance practices that threaten the privacy, safety, and civil liberties of women, immigrants, and other vulnerable Americans."
Illinois Secretary of State Alexi Giannoulias launched an audit after EFF research showed Flock allowed U.S. Customs and Border Protection to access Illinois data in violation of state privacy laws. In November, EFF partnered with the ACLU of Northern California to file a lawsuit against San Jose and its police department, challenging warrantless searches of millions of ALPR records. Between June 5, 2024 and June 17, 2025, SJPD and other California law enforcement agencies searched San Jose's database 3,965,519 times—a staggering figure illustrating the vast scope of warrantless surveillance enabled by Flock's infrastructure.
Our investigations also fueled municipal resistance to Flock Safety. Communities from Austin to Evanston to Eugene successfully canceled or refused to renew their Flock contracts after organizing campaigns centered on our research documenting discriminatory policing, immigration enforcement, threats to reproductive rights, and chilling effects on protest. These victories demonstrate that communities—armed with evidence of Flock's harms—can challenge and reject surveillance infrastructure that threatens civil liberties.
In October 2025, Flock announced plans to expand its gunshot detection microphones to listen for "human distress" including screaming. This dangerous expansion transforms audio sensors into powerful surveillance tools monitoring human voices on city streets. High-powered microphones above densely populated areas raise serious questions about wiretapping laws, false alerts, and potential for dangerous police responses to non-emergencies. After EFF exposed this feature, Flock quietly amended its marketing materials to remove explicit references to "screaming"—replacing them with vaguer language about "distress" detection—while continuing to develop and deploy the technology.
Flock Safety's surveillance infrastructure is not a neutral public safety tool. It's a system that enables and amplifies racist policing, threatens reproductive rights, and chills constitutionally protected speech. Our 2025 investigations proved it beyond doubt. As we head into 2026, EFF will continue exposing these abuses, supporting communities fighting back, and litigating for the constitutional protections that surveillance technology has stripped away.
This article is part of our Year in Review series. Read other articles about the fight for digital rights in 2025.
You might not know it, given the many headlines focused on new questions about copyright and Generative AI, but the year’s biggest copyright case concerned an old-for-the-internet question: do ISPs have to be copyright cops? After years of litigation, that question is now squarely before the Supreme Court. And if the Supreme Court doesn’t reverse a lower court’s ruling, ISPs could be forced to terminate people’s internet access based on nothing more than mere accusations of copyright infringement. This would threaten innocent users who rely on broadband for essential aspects of daily life.
This issue turns on what courts call “secondary liability,” which is the legal idea that someone can be held responsible not for what they did directly, but for what someone else did using their product or service. The case began when music companies sued Cox Communications, arguing that the ISP should be held liable for copyright infringement committed by some of its subscribers. The Court of Appeals for the Fourth Circuit agreed, adopting a “material contribution” standard for contributory copyright liability (a rule for when service providers can be held liable for the actions of users). Under that standard, providing a service that could be used for infringement is enough to create liability when a customer infringes.
The Fourth Circuit’s rule would have devastating consequences for the public. Given copyright law’s draconian penalties, ISP would be under enormous pressure to terminate accounts whenever they get an infringement notice, whether or not the actual accountholder has infringed anything: entire households, schools, libraries, or businesses that share an internet connection. These would include:
And with more than a third of Americans having only one or no broadband provider, many users would have no way to reconnect.
EFF—along with the American Library Association, the Association of Research Libraries, and Re:Create—filed an amicus brief urging the Court to reverse the Fourth Circuit’s decision, taking guidance from patent law. In the Patent Act, where Congress has explicitly defined secondary liability, there’s a different test: contributory infringement exists only where a product is incapable of substantial non-infringing use. Internet access, of course, is overwhelmingly used for lawful purposes, making it the very definition of a “staple article of commerce” that can’t be liable under the patent framework.
The Supreme Court held a hearing in the case on December 1, and a majority of the justices seemed troubled by the implications of the Fourth Circuit’s ruling. One exchange was particularly telling: asked what should happen when the notices of infringement target a university account upon which thousands of people rely, Sony’s counsel suggested the university could resolve the issue by essentially slowing internet speeds so infringement might be less appealing. It’s hard to imagine the university community would agree that research, teaching, artmaking, library services, and the myriad other activities that rely on internet access should be throttled because of the actions of a few students. Hopefully the Supreme Court won’t either.
We expect a ruling in the case in the next few months. Fingers crossed that the Court rejects the Fourth Circuit’s draconian rule.
This article is part of our Year in Review series. Read other articles about the fight for digital rights in 2025.
It's no secret that digital surveillance and other tech-enabled oppressions are acute dangers for liberation movement workers. The rising tides of tech-fueled authoritarianism and hyper-surveillance are universal themes across the various threat models we consider. EFF's Surveillance Self-Defense project is a vital antidote to these threats, but it's not all we do to help others address these concerns. Our team often receives questions, requests for security trainings, presentations on our research, and asks for general OPSEC (operations security, or, the process of applying digital privacy and information security strategies to a current workflow or process) advising. This year stood out for the sheer number and urgency of requests we fielded.
Combining efforts across our Public Interest Technology and Activism teams, we consulted with an estimated 66 groups and organizations, with at least 2000 participants attending those sessions. These engagements typically look like OPSEC advising and training, usually merging aspects of threat modeling, cybersecurity 101, secure communications practices, doxxing self-defense, and more. The groups we work with are often focused on issue-spaces that are particularly embattled at the current moment, such as abortion access, advocacy for transgender rights, and climate justice.
Our ability to offer realistic and community-focused OPSEC advice for these liberation movement workers is something we take great pride in. These groups are often under-resourced and unable to afford typical infosec consulting. Even if they could, traditional information security firms are designed to protect corporate infrastructure, not grassroots activism. Offering this assistance also allows us to stress-test the advice given in the aforementioned Surveillance Self-Defense project with real-world experience and update it when necessary. What we learn from these sessions also informs our blog posts, such as this piece on strategies for overcoming tech-enabled violence for transgender people, and this one surveying the landscape of digital threats in the abortion access movement post-Roe.
There is still much to be done. Maintaining effective privacy and security within one's work is an ongoing process. We are grateful to be included in the OPSEC process planning for so many other human-rights defenders and activists, and we look forward to continuing this work in the coming years.
This article is part of our Year in Review series. Read other articles about the fight for digital rights in 2025.
EFF’s attorneys, activists, and technologists don’t just do the hard, endless work of defending our digital civil liberties — they also spend a lot of time and effort explaining that work to the public via media interviews.
EFF had thousands of media mentions in 2025, from the smallest hyperlocal outlets to international news behemoths. Our work on street-level surveillance — the technology that police use to spy on our communities — generated a great deal of press attention, particularly regarding automated license plate readers (ALPRs). But we also got a lot of ink and airtime for our three lawsuits against the federal government: one challenging the U.S. Office of Personnel Management's illegal data sharing, a second challenging the State Department's unconstitutional "catch and revoke" program, and the third demanding that the departments of State and Justice reveal what pressure they put on app stores to remove ICE-tracking apps.
Other hot media topics included how travelers can protect themselves against searches of their devices, how protestors can protect themselves from surveillance, and the misguided age-verification laws that are proliferating across the nation and around the world, which are an attack on privacy and free expression.
On national television, Matthew Guariglia spoke with NBC Nightly News to discuss how more and more police agencies are using private doorbell cameras to surveil neighborhoods. Tori Noble spoke with ABC’s Good Morning America about the dangers of digital price tags, as well as with ABC News Live Prime about privacy concerns over OpenAI’s new web browser.
In a sampling of mainstream national media, EFF was cited 33 times by the Washington Post, 16 times by CNN, 13 times by USA Today, 12 times by the Associated Press, 11 times by NBC News, 11 times by the New York Times, 10 times by Reuters, and eight times by National Public Radio. Among tech and legal media, EFF was cited 74 times by Privacy Daily, 35 times by The Verge, 32 times by 404 Media, 32 times by The Register, 26 times by Ars Technica, 25 times by WIRED, 21 times by Law360, 21 times by TechCrunch, 20 times by Gizmodo, and 14 times by Bloomberg Law.
Abroad, EFF was cited in coverage by media outlets in nations including Australia, Bangladesh, Belgium, Canada, Colombia, El Salvador, France, Germany, India, Ireland, New Zealand, Palestine, the Philippines, Slovakia, South Africa, Spain, Trinidad and Tobago, the United Arab Emirates, and the United Kingdom.
EFF staffers spoke to the masses in their own words via op-eds such as:
And we ruled the airwaves on podcasts including:
We're grateful to all the intrepid journalists who keep doing the hard work of reporting accurately on tech and privacy policy, and we encourage them to keep reaching out to us at press@eff.org.
This article is part of our Year in Review series. Read other articles about the fight for digital rights in 2025.
Drone as first responder (DFR) adoption really took off in 2025. Though the concept has been around since 2018, this year saw more normalization of the technology, its integration into more real-time crime center structures, and the implementation of automated deployment of drones.
A DFR program features a fleet of camera-equipped drones, which can range from just a couple to dozens or more. These are deployed from a launch pad in response to 911 calls and other calls for service, sometimes operated by a drone pilot or, increasingly, autonomously directed to the call location. The appeal is the promise of increased “situational awareness” for officers headed to a call. This video offers a short explanation of DFR, and for a list of all of the cities we know use drones, including DFR programs, check out EFF’s Atlas of Surveillance.
In order to deploy a drone beyond where it can be seen, operators need to receive a waiver from the Federal Aviation Administration (FAA), and all DFR programs require this. Police departments and technology vendors have complained that the process takes too long, and in May, FAA finalized reworked requirements, leading to a flood of waiver requests. An FAA spokesperson reported that in the first two months of the new waiver process, it had approved 410 such waivers, already accounting for almost a third of the approximately 1,400 DFR waivers that had ever been granted.
The federal government made other major moves on the drone front this year. A month after the new waivers went to effect, President Trump issued an Executive Order with aspirations for advancing the country’s drone industry. And at the end of the year, one of the largest drone manufacturers in the world and one of the biggest purveyors of law enforcement drones, DJI, will be banned from launching new products in the U.S. unless the federal government conducts a security audit that was mandated by the National Defense Authorization Act. However, at the moment, it doesn’t seem like that audit will happen, and if it doesn’t, it won’t be surprising to see other drone manufacturers leveraging the ban to boost their own products.
Early iterations of drone use required a human operator, but this year, police drone companies began releasing automated flying machines that don’t require much human intervention at all. New models can rely on AI and automated directions to launch and direct a drone.
This article is part of our Year in Review series. Read other articles about the fight for digital rights in 2025.
This was the year we saw DFR integrated with other tools and tech companies teamed up to bring even more powerful surveillance. Flock Safety added automated license plate readers (ALPR) to their drones. Axon and Skydio built on the partnership they launched in 2024. Drone manufacturer Brinc teamed up with Motorola Solutions on a DFR program. Drone company Paladin teamed up with a company called SkyeBrowse to add 3-D mapping of the environment to their list of features.
DFR also is increasingly part of the police plans for real-time crime centers, meaning that the footage being captured by these flying cameras is being integrated into other streams and analyzed in ways that we’re still learning about.
Transparency around adoption, use, and oversight is always crucial, particularly when it comes to police surveillance, and EFF has been tracking the growth of DFR programs across the country. We encourage you to use your local public records laws to investigate them further. Examples of the kinds of requests and the responsive documents people have already received — including flight logs, policies, and other information — can be found on MuckRock.
Flying cameras are bad enough. They can see and record footage from a special vantage point, capturing video of your home, your backyard, and your movements that should require clear policies around retention, audits, and use, including when the cameras shouldn’t be recording. We’re also seeing that additional camera analysis and other physical features that can be added (so-called “payloads”) — like thermal cameras and even tear gas — can make drones even more powerful and that police technology companies are encouraging DFR as part of surveillance packages.
It's important that next year we all advocate for, and enforce, standards in adopting and using these DFRs. Check the Atlas to see if they are used where you live and learn more about drones and other surveillance tools on EFF’s Street-Level Surveillance Hub.
This article is part of our Year in Review series. Read other articles about the fight for digital rights in 2025.
This year, you may have heard EFF sounding off about our civil liberties on NPR, BBC Radio, or any number of podcasts. But we also started sharing our voices directly with listeners in 2025. In June, we revamped EFFector, our long-running electronic newsletter, and launched a new audio edition to accompany it.
Providing a recap of the week's most important digital rights news, EFFector's audio companion features exclusive interviews where EFF's lawyers, activists, and technologists can dig deeper into the biggest stories in privacy, free speech, and innovation. Here are just some of the best interviews from EFFector Audio in 2025.
Earlier this year, the Trump administration launched a sprawling surveillance program to spy on the social media activity of millions of noncitizens—and punish those who express views it doesn't like. This fall, EFF's Lisa Femia came onto EFFector Audio to explain how this scheme works, its impact on free speech, and, importantly, why EFF is suing to stop it.
"We think all of this is coming together as a way to chill people's speech and make it so they do not feel comfortable expressing core political viewpoints protected by the First Amendment," Femia said.
But Lisa was hardly the only guest talking about surveillance. In November, EFF's Andrew Crocker spoke to EFFector about Automated License Plate Readers (ALPRs), a particularly invasive and widespread form of surveillance. ALPR camera networks take pictures of every passing vehicle and upload the location information of millions of drivers into central databases. Police can then search these databases—typically without any judicial approval—to instantly reconstruct driver movements over weeks, months, or even years at a time.
"It really is going to be a very detailed picture of your habits over the course of a long period of time," said Crocker, explaining how ALPR location data can reveal where you work, worship, and many other intimate details about your life. Crocker also talked about a new lawsuit, filed by two nonprofits represented by EFF and the ACLU of Northern California, challenging the city of San Jose's use of ALPR searches without a warrant.
Similarly, EFF's Mario Trujillo joined EFFector in early November to discuss the legal issues and mass surveillance risks around face recognition in consumer devices.
Online privacy isn’t dead. But tech giants have tried to make protecting it as annoying as possible. To help users take back control, we celebrated Opt Out October, sharing daily privacy tips all month long on our blog. In addition to laying down some privacy basics, EFF's Thorin Klosowski talked to EFFector about how small steps to protect your data can build up into big differences.
"This is a way to kind of break it down into small tasks that you can do every day and accomplish a lot," said Klosowski. "By the end of it, you will have taken back a considerable amount of your privacy."
User privacy was the focus of a number of EFFector interviews. In July, EFF's Lena Cohen spoke about what lawmakers, tech companies, and individuals can do to fight online tracking. That same month, Matthew Guariglia talked about precautions consumers can take before bringing surveillance devices like smart doorbells into their homes.
One of the most troubling trends of 2025 was the proliferation of age verification laws, which require online services to check, estimate, or verify users’ ages. Though these mandates claim to protect children, they ultimately create harmful censorship and surveillance regimes that put everyone—adults and young people alike—at risk.
This summer, EFF's Rin Alajaji came onto EFFector Audio to explain how these laws work and why we need to speak out against them.
"Every person listening here can push back against these laws that expand censorship," she said. "We like to say that if you care about internet freedom, this fight is yours."
This was just one of several interviews about free speech online. This year, EFFector also hosted Paige Collings to talk about the chaotic rollout of the UK's Online Safety Act and Lisa Femia (again!) to discuss the abortion censorship crisis on social media.
You can hear all these episodes and future installments of EFFector's audio companion on YouTube or the Internet Archive. Or check out our revamped EFFector newsletter by subscribing at eff.org/effector!
This article is part of our Year in Review series. Read other articles about the fight for digital rights in 2025.
In 2025, elected officials across the country began treating surveillance technology purchases differently: not as inevitable administrative procurements handled by police departments, but as political decisions subject to council oversight and constituent pressure. This shift proved to be the most effective anti-surveillance strategy of the year.
Since February, at least 23 jurisdictions fully ended, cancelled, or rejected Flock Safety ALPR programs (including Austin, Oak Park, Evanston, Hays County, San Marcos, Eugene, Springfield, and Denver) by recognizing surveillance procurement as political power, not administrative routine.
For decades, cities have been caught in what researchers call "legacy procurement practices": administrative norms that prioritize "efficiency" and "cost thresholds" over democratic review.
Vendors exploit this inertia through the "pilot loophole." As Taraaz and the Collaborative Research Center for Resilience (CRCR) note in a recent report, "no-cost offers" and free trials allow police departments to bypass formal procurement channels entirely. By the time the bill comes due, the surveillance is already normalised in the community, turning a purchase decision into a "continuation of service" that is politically difficult to stop.
This bureaucracy obscures the power that surveillance vendors have over municipal procurement decisions. As Arti Walker-Peddakotla details, this is a deliberate strategy. Walker-Peddakotla details how vendors secure "acquiescence" by hiding the political nature of surveillance behind administrative veils: framing tools as "force multipliers" and burying contracts in consent agendas. For local electeds, the pressure to "outsource" government decision-making makes vendor marketing compelling. Vendors use "cooperative purchasing" agreements to bypass competitive bidding, effectively privatizing the policy-making process.
The result is a dangerous "information asymmetry" where cities become dependent on vendors for critical data governance decisions. The 2025 cancellations finally broke that dynamic.
This year, cities stopped accepting this "administrative" frame. The shift came from three converging forces: audit findings that exposed Flock's lack of safeguards, growing community organizing pressure, and elected officials finally recognizing that saying "no" to a renewal was not just an option—it was the responsible choice.
When Austin let its Flock pilot expire on July 1, the decision reflected a political judgment: constituents rejected a nationwide network used for immigration enforcement. It wasn't a debate about retention rates; it was a refusal to renew.
These cancellations were also acts of fiscal stewardship. By demanding evidence of efficacy (and receiving none) officials in Hays County, Texas and San Marcos, Texas rejected the "force multiplier" myth. They treated the refusal of unproven technology not just as activism, but as a basic fiduciary duty. In Oak Park, Illinois, trustees cancelled eight cameras after an audit found Flock lacked safeguards, while Evanston terminated its 19-camera network shortly after. Eugene and Springfield, Oregon terminated 82 combined cameras in December. City electeds have also realized that every renewal is a vote for "vendor lock-in." As EPIC warns, once proprietary systems are entrenched, cities lose ownership of their own public safety data, making it nearly impossible to switch providers or enforce transparency later.
The shift was not universal. Denver illustrated the tension when Mayor Mike Johnston overrode a unanimous council rejection to extend Flock's contract. Council Member Sarah Parady rightly identified this as "mass surveillance" imposed "with no public process." This is exactly why procurement must be reclaimed: when treated as technical, surveillance vendors control the conversation; when recognized as political, constituents gain leverage.
EFF has spent years documenting how procurement functions as a lever for surveillance expansion, from our work documenting Flock Safety's troubling data-sharing practices with ICE and federal law enforcement to our broader advocacy on surveillance technology procurement reform. The 2025 victories show that when cities understand procurement as political rather than technical, they can say no. Procurement power can be the most direct route to stopping mass surveillance.
As cities move into 2026, the lesson is clear: surveillance is a choice, not a mandate, and your community has the power to refuse it. The question isn't whether technology can police more effectively; it's whether your community wants to be policed this way. That decision belongs to constituents, not vendors.
This article is part of our Year in Review series. Read other articles about the fight for digital rights in 2025.
Defending encryption has long been a bedrock of our work. Without encryption, it's impossible to have private conversations or private data storage. This year, we’ve seen attacks on these rights from all around the world.
The European Union Council has repeatedly tried to pass a controversial message scanning proposal, known as “Chat Control,” that would require secure messaging providers to scan the contents of messages. Every time this has come up since it was first introduced in 2022, it got batted down—because no matter how you slice it, client-side scanning breaks end-to-end encryption. The Danish presidency seemed poised to succeed in passing Chat Control this year, but strong pushback from across the EU caused them to reconsider and rework their stance. In its current state, Chat Control isn’t perfect, but it at least includes strong language to protect encryption, which is good news for users.
Meanwhile, France tried to pass its own encryption-breaking legislation. Unlike Chat Control, which pushed for client-side scanning, France took a different approach: allowing so-called “ghost participants,” where law enforcement could silently join encrypted chats. Thankfully, the French National Assembly did the right thing and rejected this dangerous proposal.
It wasn’t all wins, though.
Perhaps the most concerning encryption issue is still ongoing in the United Kingdom, where the British government reportedly ordered Apple to backdoor its optional end-to-end encryption in iCloud. In response, Apple disabled one of its strongest security features, Advanced Data Protection, for U.K. users. After some back and forth with the U.S., the U.K. allegedly rewrote the demand, to clarify it was limited to only apply to British users. That doesn’t make it any better. Tribunal hearings are planned for 2026, and we’ll continue to monitor developments.
Speaking of developments to keep an eye on, the European Commission released its “Technology Roadmap on Encryption” which discusses new ways for law enforcement to access encrypted data. There’s a lot that could happen with this roadmap, but let’s be clear, here: EU officials should scrap any roadmap focused on encryption circumvention and instead invest in stronger, more widespread use of end-to-end encryption.
The U.S. had its share of battles, too. The Senate re-introduced the STOP CSAM Act, which threatened to compromise encryption by requiring encrypted communication providers to have knowledge about what sorts of content their services are being used to send. The bill allows encrypted services to raise a legal defense—but only after they’ve been sued. That's not good enough. STOP CSAM would force encryption providers to defend against costly lawsuits over content they can't see or control. And a jury could still consider the use of encryption to be evidence of wrongdoing.
In Florida, a bill ostensibly about minors' social media use also just so happened to demand a backdoor into encryption services—already an incredible overreach. It went further, attempting to ban disappearing messages and grant parents unrestricted access to their kids’ messages as well. Thankfully, the Florida Legislature ended without passing it.
It is unlikely these sorts of attempts to undermine encryption will suddenly stop. But whatever comes next, EFF will continue to stand up for everyone's right to use encryption to have secure and private online communications.
This article is part of our Year in Review series. Read other articles about the fight for digital rights in 2025.
State and federal lawmakers have introduced multiple proposals in 2025 to curtail or outright block children and teenagers from accessing legal content on the internet. These lawmakers argue that internet and social media platforms have an obligation to censor or suppress speech that they consider “harmful” to young people. Unfortunately, in many of these legislative debates, lawmakers are not listening to kids, whose experiences online are overwhelmingly more positive than what lawmakers claim.
Fortunately, EFF has spent the past year trying to make sure that lawmakers hear young people’s voices. We have also been reminding lawmakers that minors, like everyone else, have First Amendment rights to express themselves online.
These rights extend to a young person’s ability to use social media both to speak for themselves and access the speech of others online. Young people also have the right to control how they access this speech, including a personalized feed and other digestible and organized ways. Preventing teenagers from accessing the same internet and social media channels that adults use is a clear violation of their right to free expression.
On top of violating minors’ First Amendment rights, these laws also actively harm minors who rely on the internet to find community, find resources to end abuse, or access information about their health. Cutting off internet access acutely harms LGBTQ+ youth and others who lack familial or community support where they live. These laws also empower the state to decide what information is acceptable for all young people, overriding parents’ choices.
Additionally, all of the laws that would attempt to create a “kid friendly” internet and an “adults-only” internet are a threat to everyone, adults included. These mandates encourage an adoption of invasive and dangerous age-verification technology. Beyond creepy, these systems incentivize more data collection, and increase the risk of data breaches and other harms. Requiring everyone online to provide their ID or other proof of their age could block legal adults from accessing lawful speech if they don’t have the right form of ID. Furthermore, this trend infringes on people’s right to be anonymous online, and creates a chilling effect which may deter people from joining certain services or speaking on certain topics
EFF has lobbied against these bills at both the state and federal level, and we have also filed briefs in support of several lawsuits to protect the First Amendment Rights of minors. We will continue to advocate for the rights of everyone online – including minors – in the future.
This article is part of our Year in Review series. Read other articles about the fight for digital rights in 2025.
It’s been a great year to be on EFF’s membership team. There's no better feeling than hanging out with your fellow digital freedom supporters and being able to say, “Oh yeah, and we’re suing the government!” We’ve done that a lot this year—and that’s all thanks to people like you.
As a token of appreciation for supporting EFF’s mission to protect privacy and free expression online for all people, we put a lot of care into meeting the members who make our work possible. Whether it’s hosting meetups, traveling to conferences, or finding new and fun ways to explain what we’re fighting for, connecting with you is always a highlight of the job.
One of my favorite perks we offer for EFF members is exclusive invites for Speakeasy meet ups. It’s a chance for us to meet the very passionate members who fuel our work!
This year, we hosted Speakeasies across the country while making the rounds at conferences. We met supporters in Mesa, AZ during CactusCon; Pasadena, CA during SCALE; Portland, OR during BSidesPDX; New York, NY during HOPE and BSidesNYC; and Seattle, WA during our panel at the University of Washington.
Of course, we also had to host a Speakeasy in our home court—and for the first time it took place in the South Bay Area in Mountain View, CA at Hacker Dojo! There, members of EFF’s D.C. Legislative team spoke about EFF’s legislative efforts and how they’ll shape digital rights for all. We even recorded that conversation for you to watch on YouTube or the Internet Archive.
And we can’t forget about our global community! Our annual online Speakeasy brought together members around the world for a conversation and Q&A with our friends at Women in Security and Privacy (WISP) about online behavioral tracking and the data broker industry. We heard and answered great questions about pushing back on online tracking and what legislative steps we can take to strengthen privacy.
Say what you will about Vegas—nothing compares to the energy of seeing thousands of EFF supporters during the summer security conferences: BSidesLV, Black Hat USA, and DEF CON. This year over one thousand people signed up to support the digital freedom movement in just that one week.
If you’ve ever seen us at a conference, you know the drill: a table full of EFF staff frantically handing out swag, answering questions, and excitedly saying hi to everyone that stops by and supports our work. This year it was especially fun to see how many people brought their Rayhunter devices.
And of course, it wouldn’t be a trip to Vegas without EFF’s annual DEF CON Poker Tournament. This year 48 supporters and friends played for money, glory, and the future of the web—all with EFF’s very own playing cards. For the first time ever, the jellybean trophy went to the same winner two years in a row!
We ramped up our livestream series, EFFecting Change, this year with a total of six livestreams covering topics including the future of social media with guests from Mastodon, Bluesky, and Spill; EFF’s 35th Anniversary and what’s next in the fight for privacy and free speech online; and generative AI, including how to address the risks of the technology while protecting civil liberties and human rights online.
We’ve got more in store for EFFecting Change in 2026, so be sure to stay up-to-date by signing up for updates!
EFF is at the forefront of protecting users from dystopian surveillance and unjust censorship online. But we’re not the only one doing this work, and we couldn’t do it without other organizations in the space. So, every year we like to award those who are courageously championing the digital rights movement.
This year we gave out three awards: the EFF Award for Defending Digital Freedoms went to Software Freedom Law Center, India, the EFF Award for Protecting Americans’ Data went to Erie Meyer, and the EFF Award for Leading Immigration and Surveillance Litigation went to Just Futures Law. You can watch the EFF Awards here and see photos from the event too!
That doesn’t even cover all of it! We even got to celebrate 35 years of EFF in July with limited-edition challenge coins and all-new member swag—plus a livestream covering EFF’s history and what’s next for us.
Grab EFF's 35th Anniversary t-shirt when you become a member today!
As the new year approaches, I always like to look back on the bright spots—especially the joy of hanging out with this incredible community. The world can feel hectic, but connecting with supporters like you is a reminder of how much good we can build when we work together.
Many thanks to all of the EFF members who joined forces with us this year. If you’ve been meaning to join, but haven’t yet, year-end is a great time to do so.
This article is part of our Year in Review series. Read other articles about the fight for digital rights in 2025.
Across ideologically diverse communities, 2025 campaigns against automated license plate reader (ALPR) surveillance kept winning. From Austin, Texas to Cambridge, Massachusetts to Eugene, Oregon, successful campaigns combined three practical elements: a motivated political champion on city council, organized grassroots pressure from affected communities, and technical assistance at critical decision moments.
The 2025 Formula for Refusal
- Institutional Authority: Council members leveraging "procurement power"—local democracy's most underutilized tool—to say no.
- Community Mobilization: A base that refuses to debate "better policy" and demands "no cameras."
- Shared Intelligence: Local coalitions utilizing shared research on contract timelines and vendor breaches.
In 2025, organizers embraced the "ugly" win: prioritizing immediate contract cancellations over the "political purity" of perfect privacy laws. Procurement fights are often messy, bureaucratic battles rather than high-minded legislative debates, but they stop surveillance where it starts—at the checkbook. In Austin, more than 30 community groups built a coalition that forced a contract cancellation, achieving via purchasing power what policy reform often delays.
In Hays County, Texas, the victory wasn't about a new law, but a contract termination. Commissioner Michelle Cohen grounded her vote in vendor accountability, explaining: "It's more about the company's practices versus the technology." These victories might lack the permanence of a statute, but every camera turned off built a culture of refusal that made the next rejection easier. This was the organizing principle: take the practical win and build on it.
Winning campaigns didn't debate technical specifications or abstract privacy principles. They started with documented harms that surveillance enabled. EFF's research showing police used Flock's network to track Romani people with discriminatory search terms, surveil women seeking abortion care, and monitor protesters exercising First Amendment rights became the evidence organizers used to build power.
In Olympia, Washington, nearly 200 community members attended a counter-information rally outside city hall on Dec. 2. The DeFlock Olympia movement countered police department claims point-by-point with detailed citations about data breaches and discriminatory policing. By Dec. 3, cameras had been covered pending removal.
In Cambridge, the city council voted unanimously in October to pause Flock cameras after residents, the ACLU of Massachusetts, and Digital Fourth raised concerns. When Flock later installed two cameras "without the city's awareness," a city spokesperson called it a "material breach of our trust" and terminated the contract entirely. The unexpected camera installation itself became an organizing moment.
The winning formula worked because it aligned different actors around refusing vehicular mass surveillance systems without requiring everyone to become experts. Community members organized neighbors and testified at hearings, creating political conditions where elected officials could refuse surveillance and survive politically. Council champions used their institutional authority to exercise "procurement power": the ability to categorically refuse surveillance technology.
To fuel these fights, organizers leveraged technical assets like investigation guides and contract timeline analysis. This technical capacity allowed community members to lead effectively without needing to become policy experts. In Eugene and Springfield, Oregon, Eyes Off Eugene organized sustained opposition over months while providing city council members political cover to refuse. "This is [a] very wonderful and exciting victory," organizer Kamryn Stringfield said. "This only happened due to the organized campaign led by Eyes Off Eugene and other local groups."
A common misconception collapsed in 2025: that surveillance technology can only be resisted in progressive jurisdictions. San Marcos, Texas let its contract lapse after a 3-3 deadlock, with Council Member Amanda Rodriguez questioning whether the system showed "return on investment." Hays County commissioners in Texas voted to terminate. Small towns like Gig Harbor, Washington rejected proposals before deployment.
As community partners like the Rural Privacy Coalition emphasize, "privacy is a rural value." These victories came from communities with different political cultures but shared recognition that mass surveillance systems weren't worth the cost or risk regardless of zip code.
In 2025, communities no longer needed to build expertise from scratch—they could access shared investigation guides, learn from victories in neighboring jurisdictions, and connect with organizers who had won similar fights. When Austin canceled its contract, it inspired organizing across Texas. When Illinois Secretary of State's audit revealed illegal data sharing with federal immigration enforcement, Evanston used those findings to terminate 19 cameras.
The combination of different forms of power—institutional authority, community mobilization, and shared intelligence—was a defining feature of this year's most effective campaigns. By bringing these elements together, community coalitions have secured cancellations or rejections in nearly two dozen jurisdictions since February, building the infrastructure to make the next refusal easier and the movement unstoppable.
This article is part of our Year in Review series. Read other articles about the fight for digital rights in 2025.
State legislatures—from Olympia, WA, to Honolulu, HI, to Tallahassee, FL, and everywhere in between—kept EFF’s state legislative team busy throughout 2025.
We saw some great wins and steps forward this year. Washington became the eighth state to enshrine the right to repair. Several states stepped up to protect the privacy of location data, with bills recognizing your location data isn't just a pin on a map—it's a powerful tool that reveals far more than most people realize. Other state legislators moved to protect health privacy. And California passed a law making it easier for people to exercise their privacy rights under the state’s consumer data privacy law.
Several states also took up debates around how to legislate and regulate artificial intelligence and its many applications. We’ll continue to work with allies in states including California and Colorado to proposals that address the real harms from some uses of AI, without infringing on the rights of creators and individual users.
We’ve also fought some troubling bills in states across the country this year. In April, Florida introduced a bill that would have created a backdoor for law enforcement to have easy access to messages if minors use encrypted platforms. Thankfully, the Florida legislature did not pass the bill this year. But it should set off serious alarm bells for anyone who cares about digital rights. And it was just one of a growing set of bills from states that, even when well-intentioned, threaten to take a wrecking ball to privacy, expression, and security in the name of protecting young people online.
Take, for example, the burgeoning number of age verification, age gating, age assurance, and age estimation bills. Instead of making the internet safer for children, these laws can incentivize or intersect with existing systems that collect vast amounts of data to force all users—regardless of age—to verify their identity just to access basic content or products. South Dakota and Wyoming, for example, are requiring any website that hosts any sexual content to implement age verification measures. But, given the way those laws are written, that definition could include essentially any site that allows user-generated or published content without age-based gatekeeping access. That could include everyday resources such as social media networks, online retailers, and streaming platforms.
Lawmakers, not satisfied with putting age gates on the internet, are also increasingly going after VPNs (virtual private networks) to prevent anyone from circumventing these new digital walls. VPNs are not foolproof tools—and they shouldn’t be necessary to access legally protected speech—but they should be available to people who want to use them. We will continue to stand against these types of bills, not just for the sake of free expression, but to protect the free flow of information essential to a free society.
This article is part of our Year in Review series. Read other articles about the fight for digital rights in 2025.
A functioning patent system depends on one basic principle: bad patents must be challengeable. In 2025, that principle was repeatedly tested—by Congress, by the U.S. Patent and Trademark Office (USPTO), and by a small number of large patent owners determined to weaken public challenges.
Two damaging bills, PERA and PREVAIL, were reintroduced in Congress. At the same time, USPTO attempted a sweeping rollback of inter partes review (IPR), one of the most important mechanisms for challenging wrongly granted patents.
EFF pushed back—on Capitol Hill, inside the Patent Office, and alongside thousands of supporters who made their voices impossible to ignore.
The Patent Eligibility Restoration Act, or PERA, would overturn the Supreme Court’s Alice and Myriad decisions—reviving patents on abstract software ideas, and even allowing patents on isolated human genes. PREVAIL, introduced by the same main sponsors in Congress, would seriously weaken the IPR process by raising the burden of proof, limiting who can file challenges, forcing petitioners to surrender court defenses, and giving patent owners new ways to rewrite their claims mid-review.
Together, these bills would have dismantled much of the progress made over the last decade.
We reminded Congress that abstract software patents—like those we’ve seen on online photo contests, upselling prompts, matchmaking, and scavenger hunts—are exactly the kind of junk claims patent trolls use to threaten creators and small developers. We also pointed out that if PREVAIL had been law in 2013, EFF could not have brought the IPR that crushed the so-called “podcasting patent.”
EFF’s supporters amplified our message, sending thousands of messages to Congress urging lawmakers to reject these bills. The result: neither bill advanced to the full committee. The effort to rewrite patent law behind closed doors stalled out once public debate caught up with it.
Congress’ push from the outside was stymied, at least for now. Unfortunately, what may prove far more effective is the push from within by new USPTO leadership, which is working to dismantle systems and safeguards that protect the public from the worst patents.
Early in the year, the Patent Office signaled it would once again lean more heavily on procedural denials, reviving an approach that allowed patent challenges to be thrown out basically whenever there was an ongoing court case involving the same patent. But the most consequential move came later: a sweeping proposal unveiled in October that would make IPR nearly unusable for those who need it most.
2025 also marked a sharp practical shift inside the agency. Newly appointed USPTO Director John Squires took personal control of IPR institution decisions, and rejected all 34 of the first IPR petitions that came across his desk. As one leading patent blog put it, an “era of no” has been ushered in at the Patent Office.
The USPTO’s proposed rule changes would:
These changes wouldn’t “balance” the system as USPTO claims—they would make bad patents effectively untouchable. Patent trolls and aggressive licensors would be insulated, while the public would face higher costs and fewer options to fight back.
We sounded the alarm on these proposed rules and asked supporters to register their opposition. More than 4,000 of you did—thank you! Overall, more than 11,000 comments were submitted. An analysis of the comments shows that stakeholders and the public overwhelmingly oppose the proposal, with 97% of comments weighing in against it.
In those comments, small business owners described being hit with vague patents they could never afford to fight in court. Developers and open-source contributors explained that IPR is often the only realistic check on bad software patents. Leading academics, patient-advocacy groups, and major tech-community institutions echoed the same point: you cannot issue hundreds of thousands of patents a year and then block one of the only mechanisms that corrects the mistakes.
The Linux Foundation warned that the rules “would effectively remove IPRs as a viable mechanism” for developers.
GitHub emphasized the increased risk and litigation cost for open-source communities.
Twenty-two patent law professors called the proposal unlawful and harmful to innovation.
Patients for Affordable Drugs detailed the real-world impact of striking invalid pharmaceutical patents, showing that drug prices can plummet once junk patents are removed.
The USPTO now faces thousands of substantive comments. Whether the agency backs off or tries to push ahead, EFF will stay engaged. Congress may also revisit PERA, PREVAIL, or similar proposals next year. Some patent owners will continue to push for rules that shield low-quality patents from any meaningful review.
But 2025 proved something important: When people understand how patent abuse affects developers, small businesses, patients, and creators, they show up—and when they do, their actions can shape what happens next.
This article is part of our Year in Review series. Read other articles about the fight for digital rights in 2025.
As reproductive rights face growing attacks globally, access to content about reproductive healthcare and abortion online has never been more critical. The internet has essential information on topics like where and how to access care, links to abortion funds, and guidance on ways to navigate potential legal risks. Reproductive rights activists use the internet to organize and build community, and healthcare providers rely on it to distribute accurate information to people in need. And for those living in one of the 20+ states where abortion is banned or heavily restricted, the internet is often the only place to find these potentially life-saving resources.
Nonetheless, both the government and private platforms are increasingly censoring abortion-related speech, at a time when we need it most. Anti-abortion legislators are actively trying to pass laws to limit online speech about abortion, making it harder to share critical resources, discuss legal options, seek safe care, and advocate for reproductive rights. At the same time, social media platforms have increasingly cracked down on abortion-related content, leading to the suppression, shadow-banning, and outright removal of posts and accounts.
This year, we worked tirelessly to fight censorship of abortion-related information online—whether it originated from the largest social media platforms or the largest state in the U.S.
As defenders of free expression and access to information online, we have a role to play in understanding where and how this is happening, shining a light on practices that endanger these rights, and taking action to ensure they’re protected. This year, we worked tirelessly to fight censorship of abortion-related information online—whether it originated from the largest social media platforms or the largest state in the U.S.
At the start of 2025, we launched the #StopCensoringAbortion campaign to collect and spotlight the growing number of stories from users that have had abortion-related content censored by social media platforms. Our goal was to better understand how and why this is happening, raise awareness, and hold the platforms accountable.
Thanks to nearly 100 submissions from educators, advocates, clinics, researchers, and influencers around the world, we confirmed what many already suspected: this speech is being removed and restricted by platforms at an alarming rate. Across the submissions we received, we saw a pattern of over enforcement, lack of transparency, and arbitrary moderation decisions aimed at reproductive health and reproductive justice advocates.
Notably, almost none of the submissions we reviewed actually violated the platforms’ stated policies. The most common reason Meta gave for removing abortion-related content was that it violated policies on Restricted Goods and Services, which prohibit any “attempts to buy, sell, trade, donate, gift or ask for pharmaceutical drugs.” But the content being removed wasn’t selling medications. Most of the censored posts simply provided factual, educational information—content that’s expressly allowed by Meta.
In a month-long 10-part series, we broke down our findings. We examined the trends we saw, including stories of individuals and organizations who needed to rely on internal connections at Meta to get wrongfully censored posts restored, examples of account suspensions without sufficient warnings, and an exploration of Meta policies and how they are wrongly applied. We provided practical tips for users to protect their posts from being removed, and we called on platforms to adopt steps to ensure transparency, a functional appeals process, more human review of posts, and consistent and fair enforcement of rules.
Social media platforms have a First Amendment right to curate the content on their sites—they can remove whatever content they want—and we recognize that. But companies like Meta claim they care about free speech, and their policies explicitly claim to allow educational information and discussions about abortion. We think they have a duty to live up to those promises. Our #StopCensoringAbortion campaign clearly shows that this isn’t happening and underscores the urgent need for platforms to review and consistently enforce their policies fairly and transparently.
On top of platform censorship, lawmakers are trying to police what people can say and see about abortion online. So in 2025, we also fought against censorship of abortion information on the legislative front.
EFF opposed Texas Senate Bill (S.B.) 2880, which would not only outlaw the sale and distribution of abortion pills, but also make it illegal to “provide information” on how to obtain an abortion-inducing drug. Simply having an online conversation about mifepristone or exchanging emails about it could run afoul of the law.
On top of going after online speakers who create and post content themselves, the bill also targeted social media platforms, websites, email services, messaging apps, and any other “interactive computer service” simply for hosting or making that content available. This was a clear attempt by Texas legislators to keep people from learning about abortion drugs, or even knowing that they exist, by wiping this information from the internet altogether.
We laid out the glaring free-speech issues with S.B. 2880 and explained how the consequences would be dire if passed. And we asked everyone who cares about free speech to urge lawmakers to oppose this bill, and others like it. Fortunately, these concerns were heard, and the bill never became law.
Our team also spent much of the year fighting dangerous age verification legislation, often touted as “child safety” bills, at both the federal and state level. We raised the alarm on how age verification laws pose significant challenges for users trying to access critical content—including vital information about sexual and reproductive health. By age-gating the internet, these laws could result in websites requiring users to submit identification before accessing information about abortion or reproductive healthcare. This undermines the ability to remain private and anonymous while searching for abortion information online.
Abortion information saves lives, and the internet is a primary (and sometimes only) source where people can access it.
As attacks on abortion information intensify, EFF will continue to fight so that users can post, host, and access abortion-related content without fear of being silenced. We’ll keep pushing for greater accountability from social media platforms and fighting against harmful legislation aimed at censoring these vital resources. The fight is far from over, but we will remain steadfast in ensuring that everyone, regardless of where they live, can access life-saving information and make informed decisions about their health and rights.
This article is part of our Year in Review series. Read other articles about the fight for digital rights in 2025.
A tidal wave of copyright lawsuits against AI developers threatens beneficial uses of AI, like creative expression, legal research, and scientific advancement. How courts decide these cases will profoundly shape the future of this technology, including its capabilities, its costs, and whether its evolution will be shaped by the democratizing forces of the open market or the whims of an oligopoly. As these cases finished their trials and moved to appeals courts in 2025, EFF intervened to defend fair use, promote competition, and protect everyone’s rights to build and benefit from this technology.
At the same time, rightsholders stepped up their efforts to control fair uses through everything from state AI laws to technical standards that influence how the web functions. In 2025, EFF fought policies that threaten the open web in the California State Legislature, the Internet Engineering Task Force, and beyond.
Copyright lawsuits against AI developers often follow a similar pattern: plaintiffs argue that use of their works to train the models was infringement and then developers counter that their training is fair use. While legal theories vary, the core issue in many of these cases is whether using copyrighted works to train AI is a fair use.
We think that it is. Courts have long recognized that copying works for analysis, indexing, or search is a classic fair use. That principle doesn’t change because a statistical model is doing the reading. AI training is a legitimate, transformative fair use, not a substitute for the original works.
More importantly, expanding copyright would do more harm than good: while creators have legitimate concerns about AI, expanding copyright won’t protect jobs from automation. But overbroad licensing requirements risk entrenching Big Tech’s dominance, shutting out small developers, and undermining fair use protections for researchers and artists. Copyright is a tool that gives the most powerful companies even more control—not a check on Big Tech. And attacking the models and their outputs by attacking training—i.e. “learning” from existing works—is a dangerous move. It risks a core principle of freedom of expression: that training and learning—by anyone—should not be endangered by restrictive rightsholders.
In most of the AI cases, courts have yet to consider—let alone decide—whether fair use applies, but in 2025, things began to speed up.
But some cases have already reached courts of appeal. We advocated for fair use rights and sensible limits on copyright in amicus briefs filed in Doe v. GitHub, Thomson Reuters v. Ross Intelligence, and Bartz v. Anthropic, three early AI copyright appeals that could shape copyright law and influence dozens of other cases. We also filed an amicus brief in Kadrey v. Meta, one of the first decisions on the merits of the fair use defense in an AI copyright case.
How the courts decide the fair use questions in these cases could profoundly shape the future of AI—and whether legacy gatekeepers will have the power to control it. As these cases move forward, EFF will continue to defend your fair use rights.
Rightsholders also tried to make an end-run around fair use by changing the technical standards that shape much of the internet. The IETF, an Internet standards body, has been developing technical standards that pose a major threat to the open web. These proposals would give websites to express “preference signals” against certain uses of scraped data—effectively giving them veto power over fair uses like AI training and web search.
Overly restrictive preference signaling threatens a wide range of important uses—from accessibility tools for people with disabilities to research efforts aimed at holding governments accountable. Worse, the IETF is dominated by publishers and tech companies seeking to embed their business models into the infrastructure of the internet. These companies aren’t looking out for the billions of internet users who rely on the open web.
That’s where EFF comes in. We advocated for users’ interests in the IETF, and helped defeat the most dangerous aspects of these proposals—at least for now.
The AI copyright battles of 2025 were never just about compensation—they were about control. EFF will continue working in courts, legislatures, and standards bodies to protect creativity and innovation from copyright maximalists.
Age verification mandates won't magically keep young people safer online, but that has not stopped governments around the world spending this year implementing or attempting to introduce legislation requiring all online users to verify their ages before accessing the digital space.
The UK’s misguided approach to protecting young people online took many headlines due to the reckless and chaotic rollout of the country’s Online Safety Act, but they were not alone: courts in France ruled that porn websites can check users’ ages; the European Commission pushed forward with plans to test its age-verification app; and Australia’s ban on under-16s accessing social media was recently implemented.
Through this wave of age verification bills, politicians are burdening internet users and forcing them to sacrifice their anonymity, privacy, and security simply to access lawful speech. For adults, this is true even if that speech constitutes sexual or explicit content. These laws are censorship laws, and rules banning sexual content usually hurt marginalized communities and groups that serve them the most.
In response, we’ve spent this year urging governments to pause these legislative initiatives and instead protect everyone’s right to speak and access information online. Here are three ways we pushed back [against these bills] in 2025:
Banning a certain user group changes nothing about a platform’s problematic privacy practices, insufficient content moderation, or business models based on the exploitation of people’s attention and data. And assuming that young people will always find ways to circumvent age restrictions, the ones that do will be left without any protections or age-appropriate experiences.
Yet Australia’s government recently decided to ignore these dangers by rolling out a sweeping regime built around age verification that bans users under 16 from having social media accounts. In this world-first ban, platforms are required to introduce age assurance tools to block under-16s, demonstrate that they have taken “reasonable steps” to deactivate accounts used by under-16s, and prevent any new accounts being created or face fines of up to 49.5 million Australian dollars ($32 million USD). The 10 banned platforms—Instagram, Facebook, Threads, Snapchat, YouTube, TikTok, Kick, Reddit, Twitch and X—have each said they’ll comply with the legislation, leading to young people losing access to their accounts overnight.
Similarly, the European Commission this year took a first step towards mandatory age verification that could undermine privacy, expression, and participation rights for young people—rights that have been fully enshrined in international human rights law through its guidelines under Article 28 of the Digital Services Act. EFF submitted feedback to the Commission’s consultation on the guidelines, emphasizing a critical point: Mandatory age verification measures are not the right way to protect minors, and any online safety measure for young people must also safeguard their privacy and security. Unfortunately, the EU Parliament already went a step further, proposing an EU digital minimum age of 16 for access to social media, a move that aligns with EU Commission’s president Ursula von der Leyen’s recent public support for measures inspired by Australia’s model.
This year, the UK had a moment—and not a good one. In late July, new rules took effect under the Online Safety Act that now require all online services available in the UK to assess whether they host content considered harmful to children, and if so, these services must introduce age checks to prevent children from accessing such content. Online services are also required to change their algorithms and moderation systems to ensure that content defined as harmful, like violent imagery, is not shown to young people.
The UK’s scramble to find an effective age verification method shows us that there isn't one, and it’s high time for politicians to take that seriously. As we argued throughout this year, and during the passage of the Online Safety Act, any attempt to protect young people online should not include measures that require platforms to collect data or remove privacy protections around users’ identities. The approach that UK politicians have taken with the Online Safety Act is reckless, short-sighted, and will introduce more harm to the very young people that it is trying to protect.
We’re seeing these narratives and regulatory initiatives replicated from the UK to U.S. states and other global jurisdictions, and we’ll continue urging politicians not to follow the UK’s lead in passing similar legislation—and to instead explore more holistic approaches to protecting all users online.
There is not yet a legal obligation to verify users’ ages at the EU level, but policymakers and regulators are already embracing harmful age verification and age assessment measures in the name of reducing online harms.
These demands steer the debate toward identity-based solutions, such as the EU Digital Identity Wallet, which will become available in 2026. This has come with its own realm of privacy and security concerns, such as long-term identifiers (which could result in tracking) and over-exposure of personal information. Even more concerning is, instead of waiting for the full launch of the EU DID Wallet, the Commission rushed a “mini AV” app out this year ahead of schedule, citing an urgent need to address concerns about children and the harms that may come to them online.
However, this proposed solution directly tied national ID to an age verification method. This also comes with potential mission creep of what other types of verification could be done in EU member states once this is fully deployed—while the focus of the “mini AV” app is for now on verifying age, its release to the public means that the infrastructure to expand ID checks to other purposes is in place, should the government mandate that expansion in the future.
Without the proper safeguards, this infrastructure could be leveraged inappropriately—all the more reason why lawmakers should explore more holistic approaches to children's safety.
The internet is an essential resource for young people and adults to access information, explore community, and find themselves. The issue of online safety is not solved through technology alone, and young people deserve a more intentional approach to protecting their safety and privacy online—not this lazy strategy that causes more harm that it solves.
Rather than weakening rights for already vulnerable communities online, politicians must acknowledge these shortcomings and explore less invasive approaches to protect all people from online harms. We encourage politicians to look into what is best, and not what is easy; and in the meantime, we’ll continue fighting for the rights of all users on the internet in 2026.
This article is part of our Year in Review series. Read other articles about the fight for digital rights in 2025.
2025 was a stellar year for EFF’s award-winning podcast, “How to Fix the Internet,” as our sixth season focused on the tools and technology of freedom.
It seems like everywhere we turn we see dystopian stories about technology’s impact on our lives and our futures—from tracking-based surveillance capitalism, to street level government surveillance, to the dominance of a few large platforms choking innovation, to the growing efforts by authoritarian governments to control what we see and say—the landscape can feel bleak. Exposing and articulating these problems is important, but so is envisioning and then building solutions. That’s where our podcast comes in.
EFF's How to Fix the Internet podcast offers a better way forward. Through curious conversations with some of the leading minds in law and technology, EFF Executive Director Cindy Cohn and Activism Director Jason Kelley explore creative solutions to some of today’s biggest tech challenges. Our sixth season, which ran from May through September, featured:
This article is part of our Year in Review series. Read other articles about the fight for digital rights in 2025.
Earlier this year, both chambers of Congress passed the TAKE IT DOWN Act. This bill, while well-intentioned, gives powerful people a new legal tool to force online platforms to remove lawful speech that they simply don't like.
The bill, sponsored by Senate Commerce Chair Ted Cruz (R-TX) and Rep. Maria Salazar (R-FL), sought to speed up the removal of troubling online content: non-consensual intimate imagery (NCII). The spread of NCII is a serious problem, as is digitally altered NCII, sometimes called “deepfakes.” That’s why 48 states have specific laws criminalizing the distribution of NCII, in addition to the long-existing defamation, harassment, and extortion statutes—all of which can be brought to bear against those who abuse NCII. Congress can and should protect victims of NCII by enforcing and improving these laws.
Unfortunately, TAKE IT DOWN takes another approach: it creates an unneeded notice-and-takedown system that threatens free expression, user privacy, and due process, without meaningfully addressing the problem it seeks to solve.
While Congress was still debating the bill, EFF, along with the Center for Democracy & Technology (CDT), Authors Guild, Demand Progress Action, Fight for the Future, Freedom of the Press Foundation, New America’s Open Technology Institute, Public Knowledge, Restore The Fourth, SIECUS: Sex Ed for Social Change, TechFreedom, and Woodhull Freedom Foundation, sent a letter to the Senate outlining our concerns with the proposal.
First, TAKE IT DOWN’s removal provision applies to a much broader category of content—potentially any images involving intimate or sexual content—than the narrower NCII definitions found elsewhere in the law. We worry that bad-faith actors will use the law’s expansive definition to remove lawful speech that is not NCII and may not even contain sexual content.
Worse, the law contains no protections against frivolous or bad-faith takedown requests. Lawful content—including satire, journalism, and political speech—could be wrongly censored. The law requires that apps and websites remove content within 48 hours or face significant legal risks. That ultra-tight deadline means that small apps or websites will have to comply so quickly to avoid legal risk, that they won’t be able to investigate or verify claims.
Finally, there are no legal protections for providers when they believe a takedown request was sent in bad faith to target lawful speech. TAKE IT DOWN is a one-way censorship ratchet, and its fast timeline discourages providers from standing up for their users’ free speech rights.
This new law could lead to the use of automated filters that tend to flag legal content, from commentary to news reporting. Communications providers that offer users end-to-end encrypted messaging, meanwhile, may be served with notices they simply cannot comply with, given the fact that these providers can’t view the contents of messages on their platforms. Platforms could respond by abandoning encryption entirely in order to be able to monitor content, turning private conversations into surveilled spaces.
We asked for several changes to protect legitimate speech that is not NCII, and to include common-sense safeguards for encryption. Thousands of EFF members joined us by writing similar messages to their Senators and Representatives. That resulted in several attempts to offer common-sense amendments during the Committee process.
However, Congress passed the bill without those needed changes, and it was signed into law in May 2025. The main takedown provisions of the bill will take effect in 2026. We’ll be pushing online platforms to be transparent about the content they take down because of this law, and will be on the watch for takedowns that overreach and censor lawful speech.
This article is part of our Year in Review series. Read other articles about the fight for digital rights in 2025.
The world has been forced to bear the weight of billionaires and politicians who salivate over making tech more invasive, more controlling, and more hostile. That's why EFF’s mission for your digital rights is crucial, and why your support matters more than ever. You can fuel the fight for privacy and free speech with as little as $5 or $10 a month:
Become a Monthly Sustaining Donor
When you donate by December 31, your monthly support goes even further by unlocking bonus Year-End Challenge grants! With your help, EFF can receive up to seven grants that increase in size as the number of supporters grows (check our progress on the counter). Many thanks to EFF’s Board of Directors for creating the 2025 challenge fund.
The EFF team makes every dollar count. EFF members giving just $10 or less each month raised $400,000 for digital rights in the last year. That funds court motions, software development, educational campaigns, and investigations for the public good every day. EFF member support matters, and we need you.
You can help EFF hold corporations and authoritarians to account. We fight for tech users in the courts and we lobby and educate lawmakers, all while developing free privacy-enhancing tech and educational resources so people can protect themselves now. Your monthly donation will keep us going strong in this pivotal moment.
Get your choice of free gear when you join EFF!
Your privacy online and the right to express yourself are powerful—and it’s the reason authoritarians work so viciously to take them away. But together, we can make sure technology remains a tool for the people. Become a monthly Sustaining Donor or give a one-time donation of any size by December 31 and unlock additional Year-End Challenge grants!
Unlock Year-End Challenge Grants
EFF Members have carried the movement for privacy and free expression for decades. You can help move the mission even further! Here’s some sample language that you can share with your networks:
We need to stand together and ensure technology works for us, not against us. Donate any amount to EFF by Dec 31, and you'll help unlock challenge grants! https://eff.org/yec
Bluesky | Facebook | LinkedIn | Mastodon
(more at eff.org/social)
_________________
EFF is a member-supported U.S. 501(c)(3) organization. We’re celebrating TWELVE YEARS of top ratings from the nonprofit watchdog Charity Navigator! Your donation is tax-deductible as allowed by law.
In 2024, EFF wrote our initial blog about what could go wrong when police let AI write police reports. Since then, the technology has proliferated at a disturbing rate. Why? The most popular generative AI tool for writing police reports is Axon’s Draft One, and Axon also happens to be the largest provider of body-worn cameras to police departments in the United States. As we’ve written, companies are increasingly bundling their products to make it easier for police to buy more technology than they may need or that the public feels comfortable with.
We have good news and bad news.
Here’s the bad news: AI written police reports are still unproven, untransparent, and downright irresponsible–especially when the criminal justice system, informed by police reports, is deciding people’s freedom. The King County prosecuting attorney’s office in Washington state barred police from using AI to write police reports. As their memo read, “We do not fear advances in technology – but we do have legitimate concerns about some of the products on the market now... AI continues to develop and we are hopeful that we will reach a point in the near future where these reports can be relied on. For now, our office has made the decision not to accept any police narratives that were produced with the assistance of AI.”
In July of this year, EFF published a two-part report on how Axon designed Draft One to defy transparency. Police upload their body-worn camera’s audio into the system, the system generates a report that the officer is expected to edit, and then the officer exports the report. But when they do that, Draft One erases the initial draft, and with it any evidence of what portions of the report were written by AI and what portions were written by an officer. That means that if an officer is caught lying on the stand – as shown by a contradiction between their courtroom testimony and their earlier police report – they could point to the contradictory parts of their report and say, “the AI wrote that.” Draft One is designed to make it hard to disprove that.
In this video of a roundtable discussion about Draft One, Axon’s senior principal product manager for generative AI is asked (at the 49:47 mark) whether or not it’s possible to see after-the-fact which parts of the report were suggested by the AI and which were edited by the officer. His response (bold and definition of RMS added):
“So we don’t store the original draft and that’s by design and that’s really because the last thing we want to do is create more disclosure headaches for our customers and our attorney’s offices—so basically the officer generates that draft, they make their edits, if they submit it into our Axon records system then that’s the only place we store it, if they copy and paste it into their third-party RMS [records management system] system as soon as they’re done with that and close their browser tab, it’s gone. It’s actually never stored in the cloud at all so you don’t have to worry about extra copies floating around.”
Yikes!
All of this obfuscation also makes it incredibly hard for people outside police departments to figure out if their city’s officers are using AI to write reports–and even harder to use public records requests to audit just those reports. That’s why this year EFF also put out a comprehensive guide to help the public make their records requests as tailored as possible to learn about AI-generated reports.
Ok, now here’s the good news: People who believe AI-written police reports are irresponsible and potentially harmful to the public are fighting back.
This year, two states have passed bills that are an important first step in reigning in AI police reports. Utah’s SB 180 mandates that police reports created in whole or in part by generative AI have a disclaimer that the report contains content generated by AI. It also requires officers to certify that they checked the report for accuracy. California’s SB 524 went even further. It requires police to disclose, on the report, if it was used to fully or in part author a police report. Further, it bans vendors from selling or sharing the information a police agency provided to the AI. The bill also requires departments to retain the first draft of the report so that judges, defense attorneys, or auditors could readily see which portions of the final report were written by the officer and which portions were written by the computer.
In the coming year, anticipate many more states joining California and Utah in regulating, or perhaps even banning, police from using AI to write their reports.
This article is part of our Year in Review series. Read other articles about the fight for digital rights in 2025.
The US legal profession was just one of the pillars of American democracy that was targeted in the early days of the second Trump administration. At EFF, we were proud to publicly and loudly support the legal profession and, most importantly, continue to do our work challenging the government’s erosion of digital rights—work that became even more critical as many law firms shied away from pro bono work.
For those that don’t know: pro bono work is work that for-profit law firms undertake for the public good. This usually means providing legal counsel to clients who desperately need but cannot afford it. It’s a vital practice, since non-profits like EFF don’t have the same capacity, resources, or expertise of a classic white shoe law firm. It’s mutually beneficial, actually, since law firms and non-profits have different experience and areas of expertise that can supplement each other’s work.
A little more than a month into the new administration, President Trump began retaliating against large law firms who supported had investigations against him or litigated against his interests, representing clients either challenging his policies during his first term or defending the outcome of the 2020 election among other cases. The retaliation quickly spread to other firms—firms lost government contracts and had security clearances stripped from their lawyers. Twenty large law firm were threatened by the Equal Employment Opportunity Commission over their DEI policies. Individual lawyers were also targeted. The policy attacking the legal profession was memorialized as official policy in the March 22, 2025 presidential memo Preventing Abuses of the Legal System and the Federal Court.
Although many of the targeted firms shockingly and regrettably capitulated, a few law firms sued to undo the actions against them. EFF was eager to support them, joining amicus briefs in each case. Over 500 law firms across the country joined supportive amicus briefs as well.
We also thought it critically important to publicly state our support for the targeted law firms and to call out the administration’s actions as violating the rule of law. So we did. We actually expected numerous law firms and legal organizations to also issue statements. But no one else did. EFF was thus the very first non-targeted legal organization in the country, either law firm or nonprofit, to publicly oppose the administration’s attack on the independence of the legal profession. Fortunately, within the week, firms started to speak up as well. As did the American Bar Association.
In the meantime, EFF’s legal work has become even more critical as law firms have reportedly pulled back on their pro bono hours since the administration’s attacks. Indeed, recognizing the extraordinary need, we ramped up out litigation, including cases against the federal government, suing DOGE for stealing Americans’ data, the state department for chilling visa-holders’ speech by surveilling and threatening to surveil their social media posts, and seeking records of the administration’s demands to online platforms to remove ICE oversight apps.
And we’re going to keep on going in 2026 and beyond.
This article is part of our Year in Review series. Read other articles about the fight for digital rights in 2025.
Each December we take a moment to publish a series of blog posts that look back at the things we’ve accomplished in fighting for your rights and privacy the past 12 months. But this year I’ve been thinking not just about the past 12 months, but also over the past 25 years I’ve spent at EFF. As many folks know, I’ve decided to pass the leadership torch and will leave EFF in 2026, so this will be the last time I write one of these annual reviews. It’s bittersweet, but I’m filled with pride, especially about how we stick with fights over the long run.
EFF has come a long way since I joined in 2000. In so many ways, the work and reputation we have built laid the groundwork for years like 2025 – when freedom, justice and innovation were under attack from many directions at once with tech unfortunately at the center of many of them. As a result, we launched our Take Back CRTL campaign to put the focus on fighting back.
In addition to the specific issues we address in this year-end series of blog posts, EFF brought our legal expertise to several challenges to the Trump Administration’s attacks on privacy, free speech and security, including directly bringing two cases against the government and filing multiple amicus briefs in others. In some ways, that’s not new: we’ve worked in the courts to hold the government accountable all the way back to our founding in1990.
In this introductory blog post, however, I want to highlight two topics that attest to our long history of advocacy. The first is our battle against the censorship and privacy nightmares that come from requirements that internet users to submit to age verification. We’ve long known that age verification technologies, which aim to block young people from viewing or sharing information that the government deems “harmful” or “offensive,” end up becoming tools of censorship. They often rely on facial recognition and other techniques that have unacceptable levels of inaccuracy and that create security risks. Ultimately, they are surveillance systems that chill access to vital online communities and resources, and burden the expressive rights of adults and young people alike.
The second is automated license plate readers (ALPR), which serve as a mass surveillance network of our locations as we go about our day. We sued over this technology in 2013, demanding public access to records about their use and ultimately won at the California Supreme Court. But 2025 is the year that the general public began to understand just how much information is being collected and used by governments and private entities alike, and to recognize the dangers that causes. Our investigations team filed another public records requests, revealing racist searches done by police. And 12 years later after our first lawsuit, our lawyers filed another case, this time directly challenging the ALPR policies of San Jose, California. In addition, our activists have been working with people in municipalities across the country who want to stop their city’s use of ALPR in their communities. Groups in Austin, Texas, for example, worked hard to get their city to reject a new contract for these cameras.
These are just two issues of many that have engaged our lawyers, activists, and technologists this year. But they show how we dig in for the long run and are ready when small issues become bigger ones.
The more than 100 people who work at EFF spent this last year proving their mettle in battles, many of which are nowhere near finished. But we will push on, and when those issues breach public consciousness, we’ll be ready.
We can only keep doggedly working on these issues year after year because of you, our members and supporters. You engage on these issues, you tell us when something is happening in your town, and your donations power everything we do. This may be my last end-of-the-year blog post, but thanks to you, EFF is here to stay. We’re strong, we’re ready, and we know how to stick with things for the long run. Thanks for holding us up.
This article is part of our Year in Review series. Read other articles about the fight for digital rights in 2025.
Since earliest days of computer games, people have tinkered with the software to customize their own experiences or share their vision with others. From the dad who changed the game’s male protagonist to a girl so his daughter could see herself in it, to the developers who got their start in modding, games have been a medium where you don’t just consume a product, you participate and interact with culture.
For decades, that participatory experience was a key part of one of the longest-running video games still in operation: Everquest. Players had the official client, acquired lawfully from EverQuest’s developers, and modders figured out how to enable those clients to communicate with their own servers and then modify their play experience – creating new communities along the way.
Everquest’s copyright owners implicitly blessed all this. But the current owners, a private equity firm called Daybreak, want to end that independent creativity. They are using copyright claims to threaten modders who wanted to customize the EverQuest experience to suit a different playstyle, running their own servers where things worked the way they wanted.
One project in particular is in Daybreak’s crosshairs: “The Hero’s Journey” (THJ). Daybreak claims THJ has infringed its copyrights in Everquest visuals and character, cutting into its bottom line.
Ordinarily, when a company wants to remedy some actual harm, its lawyers will start with a cease-and-desist letter and potentially pursue a settlement. But if the goal is intimidation, a rightsholder is free to go directly to federal court and file a complaint. That’s exactly what Daybreak did, using that shock-and-awe approach to cow not only The Hero’s Journey team, but unrelated modders as well.
Daybreak’s complaint seems to have dazzled the judge in the case by presenting side-by-side images of dragons and characters that look identical in the base game and when using the mod, without explaining that these images are the ones provided by EverQuest’s official client, which players have lawfully downloaded from the official source. The judge wound up short-cutting the copyright analysis and issuing a ruling that has proven devastating to the thousands of players who are part of EverQuest modding communities.
Daybreak and the developers of The Hero’s Journey are now in private arbitration, and Daybreak has wasted no time in sending that initial ruling to other modders. The order doesn’t bind anyone who’s unaffiliated with The Hero’s Journey, but it’s understandable that modders who are in it for fun and community would cave to the implied threat that they could be next.
As a result, dozens of fan servers have stopped operating. Daybreak has also persuaded the maintainers of the shared server emulation software that most fan servers rely upon, EQEmulator, to adopt terms of service that essentially ban any but the most negligible modding. The terms also provide that “your operation of an EQEmulator server is subject to Daybreak’s permission, which it may revoke for any reason or no reason at any time, without any liability to you or any other person or entity. You agree to fully and immediately comply with any demand from Daybreak to modify, restrict, or shut down any EQEmulator server.”
This is sadly not even an uncommon story in fanspaces—from the dustup over changes to the Dungeons and Dragons open gaming license to the “guidelines” issued by CBS for Star Trek fan films, we see new generations of owners deciding to alienate their most avid fans in exchange for more control over their new property. It often seems counterintuitive—fans are creating new experiences, for free, that encourage others to get interested in the original work.
Daybreak can claim a shameful victory: it has imposed unilateral terms on the modding community that are far more restrictive than what fair use and other user rights would allow. In the process, it is alienating the very people it should want to cultivate as customers: hardcore Everquest fans. If it wants fans to continue to invest in making its games appeal to broader audiences and serve as testbeds for game development and sources of goodwill, it needs to give the game’s fans room to breathe and to play.
If you’ve been a target of Daybreak’s legal bullying, we’d love to hear from you; email us at info@eff.org.
Interviewer: Jillian York
Sami Ben Gharbia is a Tunisian human rights campaigner, blogger, writer and freedom of expression advocate. He founded Global Voices Advocacy, and is the co-founder and current publisher of the collective media organization Nawaat, which won the EFF Award in 2011.
Jillian York: So first, what is your personal definition, or how do you conceptualize freedom of expression?
Sami Ben Gharbia: So for me, freedom of expression, it is mainly as a human. Like, I love the definition of Arab philosophers to human beings, we call it “speaking animal”. So that's the definition in logic, like the science of logic, meditated on by the Greeks, and that defines a human being as a speaking animal, which means later on. Descartes, the French philosopher, describes it like the Ergo: I think, so I am. So the act of speaking is an act of thinking, and it's what makes us human. So this is my definition that I love about freedom of expression, because it's the condition, the bottom line of our human being.
JY: I love that. Is that something that you learned about growing up?
SBG: You mean, like, reading it or living?
JY: Yeah, how did you come to this knowledge?
SBG: I read a little bit of logics, like science of logic, and this is the definition that the Arabs give to define what is a human being; to differentiate us from, from plants or animals, or, I don't know, rocks, et cetera. So the humans are speaking, animals,
JY: Oh, that's beautiful.
SBG: And by speaking, it's in the Arabic definition of the word speaking, it's thinking. It's equal to thinking.
JY: At what point, growing up, did you realize…what was the turning point for you growing up in Tunisia and realizing that protecting freedom of expression was important?
SBG: Oh, I think, I was born in 1967 and I grew up under an authoritarian regime of the “father” of this Tunisian nation, Bourghiba, the first president of Tunisia, who got us independence from France. And during the 80s, it was very hard to find even books that speak about philosophy, ideology, nationalism, Islamism, Marxism, etc. So to us, almost everything was forbidden. So you need to hide the books that you smuggle from France or from libraries from other cities, et cetera. You always hide what you are reading because you do not want to expose your identity, like you are someone who is politically engaged or an activist. So, from that point, I realized how important freedom of expression is, because if you are not allowed even to read or to buy or to exchange books that are deemed to be controversial or are so politically unacceptable under an authoritarian regime, that's where the fight for freedom of expression should be at the forefront of of any other fights. That's the fight that we need to engage in in order to secure other rights and freedoms.
JY: You speak a number of languages, at what point did you start reading and exploring other languages than the one that you grew up speaking?
SBG: Oh, I think, well, we learn Arabic, French and English in school, and like, primary school, secondary school, so these are our languages that we take from school and from our readings, etc, and interaction with other people in Tunisia. But my first experience living in a country that speaks another language that I didn't know was in Iran. So I spent, in total, one and a half years there in Iran, where I started to learn a fourth language that I really intended to use. It's not a Latin language. It is a special language, although they use almost the same letters and alphabet with some difference in pronunciation and writing, but but it was easy for an Arab speaking native Tunisian to learn Farsi due to the familiarity with the alphabets and familiarity with the pronunciation of most of the alphabet itself. So, that's the first case where I was confronted with a foreign language. It was Iran. And then during my exile in the Netherlands, I was confronted by another family of languages, which is Dutch from the family of Germanic languages, and that's the fifth language that I learned in the Netherlands.
JY: Wow. And how do you feel that language relates to expression? For you?
SBG: I mean…language, it's another word. It's another universe. Because language carries culture, carries knowledge, carries history, customs. So it's a universe that is living. And once you learn to speak a new language, actually, you embrace another culture. You are more open in the way of understanding and accepting differences between other cultures, and I think that's how it makes your openness much more elastic. Like you accept other cultures more, other identities, and then you are not afraid anymore. You're not scared anymore from other identities, let's say, because I think the problem of civilization and crisis or conflict starts from ignorance—like we don't know the others, we don't know the language, we don't know the customs, the culture, the heritage, the history. That's why we are scared of other people. So the language is the first, let's say, window to other identity and acceptance of other people
JY: And how many languages do you speak now?
SBG: Oh, well, I don't know. Five for sure, but since I moved to exile a second time now, to Spain, I started learning Spanish, and I've been traveling a lot in Italy, started learning some some Italian, but it is confusing, because both are Latin languages, and they share a lot of words, and so it is confusing, but it is funny. I'm not that young to learn quickly, but I'm 58 years old, so it's not easy for someone my age to learn a new language quickly, especially when you are confused about languages from the same family as Latin.
JY: Oh, that's beautiful, though. I love that. All right, now I want to dig into the history of [2011 EFF Award winner] Nawaat. How did it start?
SBG: So Nawaat started as a forum, like in the early 2000s, even before the phenomena of blogs. Blogs started later on, maybe 2003-4, when they became the main tools for expression. Before that, we had forums where people debate ideas, anything. So it started as a forum, multiple forums hosted on the same domain name, which is Nawaat.org and little by little, we adopted new technology. We moved it. We migrated the database from from the forum to CMS, built a new website, and then we started building the website or the blog as a collective blog where people can express themselves freely, and in a political context where, similar to many other countries, a lot of people express themselves through online platforms because they are not allowed to express themselves freely through television or radio or newspaper or magazines in in their own country.
So it started mainly as an exiled media. It wasn't journalistically oriented or rooted in journalism. It was more of a platform to give voices to the diaspora, mainly the exiled Tunisian diaspora living in exile in France and in England and elsewhere. So we published Human Rights Reports, released news about the situation in Tunisia. We supported the opposition in Tunisia. We produced videos to counter the propaganda machine of the former President Ben Ali, etc. So that's how it started and evolved little by little through the changing in the tech industry, from forums to blogs and then to CMS, and then later on to to adopt social media accounts and pages. So this is how it started and why we created it that like that was not my decision. It was a friend of mine, we were living in exile, and then we said, “why not start a new platform to support the opposition and this movement in Tunisia?” And that's how we did it at first, it was fun, like it was something like it was a hobby. It wasn't our work. I was working somewhere else, and he was working something else. It was our, let's say hobby or pastime. And little by little, it became our, our only job, actually.
JY: And then, okay, so let's come to 2011. I want to hear now your perspective 14 years later. What role do you really feel that the internet played in Tunisia in 2011?
SBG: Well, it was a hybrid tool for liberation, etc. We know the context of the internet freedom policy from the US we know, like the evolution of Western interference within the digital sphere to topple governments that are deemed not friendly, etc. So Tunisia was like, a friend of the West, very friendly with France and the United States and Europe. They loved the dictatorship in Tunisia, in a way, because it secured the border. It secured the country from, by then, the Islamist movement, et cetera. So the internet did play a role as a platform to spread information and to highlight the human rights abuses that are taking place in Tunisia and to counter the narrative that is being manipulated then by the government agency, state agency, public broadcast channel, television news agency, etc.
And I think we managed it like the big impact of the internet and the blogs by then and platforms like now. We adopted English. It was the first time that the Tunisian opposition used English in its discourse, with the objective to bridge the gap between the traditional support for opposition and human rights in Tunisia that was mainly was coming from French NGOs and human rights organization towards international support, and international support that is not only coming from the traditional, usual suspects of Human Rights Watch, Amnesty International, Freedom House, et cetera. Now we wanted to broaden the spectrum of the support and to reach researchers, to reach activists, to reach people who are writing about freedom elsewhere. So we managed to break the traditional chain of support between human rights movements or organizations and human rights activists in Tunisia, and we managed to broaden that and to reach other people, other audiences that were not really touching what was going on in Tunisia, and I think that's how the Internet helped in the field of international support to the struggle in Tunisia and within Tunisia.
The impact was, I think, important to raise awareness about human rights abuses in the country, so people who are not really politically knowledgeable about the situation due to the censorship and due to the problem of access to information which was lacking in Tunisia, the internet helped spread the knowledge about the situation and help speed the process of the unrest, actually. So I think these are the two most important impacts within the country, to broaden the spectrum of the people who are reached and targeted by the discourse of political engagement and activism, and the second is to speed the process of consciousness and then the action in the street. So this is how I think the internet helped. That's great, but it wasn't the main tool. I mean, the main tool was really people on the ground and maybe people who didn't have access to the internet at all.
JY: That makes sense. So what about the other work that you were doing around that time with the Arabloggers meetings and Global Voices and the Arab Techies network. Tell us about that.
SBG: Okay, so my position was the founding director of Global Voices Advocacy, I was hired to found this, this arm of advocacy within Global Voices. And that gave me the opportunity to understand other spheres, linguistic spheres, cultural spheres. So it was beyond Tunisia, beyond the Arab world and the region. I was in touch with activists from all over the world. I mean by activists, I mean digital activists, bloggers that are living in Latin America or in Asia or in Eastern Europe, et cetera, because one of the projects that I worked on was Threatened Voices, which was a map of all people who were targeted because of their online activities. That gave me the opportunity to get in touch with a lot of activists.
And then we organized the first advocacy meeting. It was in Budapest, and we managed to invite like 40 or 50 activists from all over the world, from China, Hong Kong, Latin America, the Arab world, Eastern Europe, and Africa. And that broadened my understanding of the freedom of expression movement and how technology is being used to foster human rights online, and then the development of blog aggregators in the world, and mainly in the Arab world, like, each country had its own blog aggregator. That helped me understand those worlds, as did Global Voices. Because Global Voices was bridging the gap between what is being written elsewhere, through the translation effort of Global Voices to the English speaking world and vice versa, and the role played by Global Voices and Global Voices Advocacy made the space and the distance between all those blogospheres feel very diminished. We were very close to the blogosphere movement in Egypt or in Morocco or in Syria and elsewhere.
And that's how, Alaa Abd El Fattah and Manal Bahey El-Din Hassan and myself, we started thinking about how to establish the Arab Techies collective, because the needs that we identified—there was a gap. There was a lack of communication between pure techies, people who are writing code, building software, translating tools and even online language into Arabic, and the people who are using those tools. The bloggers, freedom of expression advocates, et cetera. And because there are some needs that were not really met in terms of technology, we thought that bringing these two words together, techies and activists would help us build new tools, translate new tools, make tools available to the broader internet activists. And that's how the Arab Techies collective was born in Cairo, and then through organizing the Arabloggers meetings two times in Beirut, and then the third in Tunisia, after the revolution.
It was a momentum for us, because it, I think it was the first time in Beirut that we brought bloggers from all Arab countries, like it was like a dream that was really unimaginable but at a certain point, but we made that happen. And then what they call the Arab revolution happened, and we lost contact with each other, because everybody was really busy with his or her own country's affairs. So Ali was really fully engaged in Egypt myself, I came back to Tunisia and was fully engaged in Tunisia, so we lost contact, because all of us were having a lot of trouble in their own country. A lot of those bloggers, like who attended the Arab bloggers meetings, few of them were arrested, few of them were killed, like Bassel was in prison, people were in exile, so we lost that connection and those conferences that brought us together, but then we've seen SMEX like filling that gap and taking over the work that started by the Arab techies and the Arab bloggers conference.
JY: We did have the fourth one in 2014 in Amman. But it was not the same. Okay, moving forward, EFF recently published this blog post reflecting on what had just happened to Nawaat, when you and I were in Beirut together a few weeks ago. Can you tell me what happened?
SBG: What happened is that they froze the work of Nawaat. Legally, although the move wasn't legal, because for us, we were respecting the law in Tunisia. But they stopped the activity of Nawaat for one month. And this is according to an article from the NGO legal framework, that the government can stop the work of an NGO if the NGO doesn't respect certain legal conditions; for them Nawaat didn't provide enough documentation that was requested by the government, which is a total lie, because we always submit all documentation on time to the government. So they stopped us from doing our job, which is what we call in Tunisia, an associated media.
It's not a company, it's not a business. It's not a startup. It is an NGO that is managing the website and the media, and now it has other activities, like we have the online website, the main website, but we also have a festival, which is a three day festival in our headquarters. We have offline debates. We bring actors, civil society, activists, politicians, to discuss important issues in Tunisia. We have a quality print magazine that is being distributed and sold in Tunisia. We have an innovation media incubation program where we support people to build projects through journalism and technology. So we have a set of offline projects that stopped for a month, and we also stopped publishing anything on the website and all our social media accounts. And now what? It's not the only one. They also froze the work of other NGOs, like the Tunisian Association of Democratic Women, which is really giving support to women in Tunisia. Also the Tunisian Forum for Social and Economic Rights, which is a very important NGO giving support to grassroots movements in Tunisia. And they stopped Aswat Nissa, another NGO that is giving support to women in Tunisia. So they targeted impactful NGOs.
So now what? It's not an exception, and we are very grateful to the wave of support that we got from Tunisian fellow citizens, and also friendly NGOs like EFF and others who wrote about the case. So this is the context in which we are living, and we are afraid that they will go for an outright ban of the network in the future. This is the worst case scenario that we are preparing ourselves for, and we might face this fate of seeing it close its doors and stop all offline activities that are taking place in Tunisia. Of course, the website will remain. We need to find a way to keep on producing, although it will really be risky for our on-the-ground journalists and video reporters and newsroom team, but we need to find a solution to keep the website alive. As an exiled media it's a very probable scenario and approach in the future, so we might go back to our exile media model, and we will keep on fighting.
JY: Yes, of course. I'm going to ask the final question. We always ask who someone’s free speech hero is, but I’m going to frame it differently for you, because you're somebody who influenced a lot of the way that I think about these topics. And so who's someone that has inspired you or influenced your work?
SBG: Although I started before the launch of WikiLeaks, for me Julian Assange was the concretization of the radical transparency movement that we saw. And for me, he is one of the heroes that really shaped a decade of transparency journalism and impacted not only the journalism industry itself, like even the established and mainstream media, such as the New York Times, Washington Post, Der Spiegel, et cetera. Wikileaks partnered with big media, but not only with big media, also with small, independent newsrooms in the Global South. So for me, Julian Assange is an icon that we shouldn't forget. And he is an inspiration in the way he uses technology to to fight against big tech and state and spy agencies and war crimes.
Fair use is not just an excuse to copy—it’s a pillar of online speech protection, and disregarding it in order to lash out at a critic should have serious consequences. That’s what we told a federal court in Channel 781 News v. Waltham Community Access Corporation, our case fighting copyright abuse on behalf of citizen journalists.
Waltham Community Access Corporation (WCAC), a public access cable station in Waltham, Massachusetts, records city council meetings on video. Channel 781 News (Channel 781), a group of volunteers who report on the city council, curates clips from those recordings for its YouTube channel, along with original programming, to spark debate on issues like housing and transportation. WCAC sent a series of takedown notices under the Digital Millennium Copyright Act (DMCA), accusing Channel 781 of copyright infringement. That led to YouTube deactivating Channel 781’s channel just days before a critical municipal election. Represented by EFF and the law firm Brown Rudnick LLP, Channel 781 sued WCAC for misrepresentations in its takedown notices under an important but underutilized provision of the DMCA.
The DMCA gives copyright holders a powerful tool to take down other people’s content from platforms like YouTube. The “notice and takedown” process requires only an email, or filling out a web form, in order to accuse another user of copyright infringement and have their content taken down. And multiple notices typically lead to the target’s account being suspended, because doing so helps the platform avoid liability. There’s no court or referee involved, so anyone can bring an accusation and get a nearly instantaneous takedown.
Of course, that power invites abuse. Because filing a DMCA infringement notice is so easy, there’s a temptation to use it at the drop of a hat to take down speech that someone doesn’t like. To prevent that, before sending a takedown notice, a copyright holder has to consider whether the use they’re complaining about is a fair use. Specifically, the copyright holder needs to form a “good faith belief” that the use is not “authorized by the law,” such as through fair use.
WCAC didn’t do that. They didn’t like Channel 781 posting short clips from city council meetings recorded by WCAC as a way of educating Waltham voters about their elected officials. So WCAC fired off DMCA takedown notices at many of Channel 781’s clips that were posted on YouTube.
WCAC claims they considered fair use, because a staff member watched a video about it and discussed it internally. But WCAC ignored three of the four fair use factors. WCAC ignored that their videos had no creativity, being nothing more than records of public meetings. They ignored that the clips were short, generally including one or two officials’ comments on a single issue. They ignored that the clips caused WCAC no monetary or other harm, beyond wounded pride. And they ignored facts they already knew, and that are central to the remaining fair use factor: by excerpting and posting the clips with new titles, Channel 781 was putting its own “spin” on the material - in other words, transforming it. All of these facts support fair use.
Instead, WCAC focused only on the fact that the clips they targeted were not altered further or put into a larger program. Looking at just that one aspect of fair use isn’t enough, and changing the fair use inquiry to reach the result they wanted is hardly the way to reach a “good faith belief.”
That’s why we’re asking the court to rule that WCAC’s conduct violated the law and that they should pay damages. Copyright holders need to use the powerful DMCA takedown process with care, and when they don’t, there needs to be consequences.
Update: Cheers to the EFF members who have allowed us to win 7 bonus grants in our 2025 Year-End Challenge! Donate now to defend privacy and free speech all year long.
What a year it’s been. We’ve seen technology unfortunately misused to supercharge the threats facing democracy: dystopian surveillance, attacks on encryption, and government censorship. These aren’t abstract dangers. They’re happening now, to real people, in real time.
EFF’s lawyers, technologists, and activists are pushing back. But we need you in this fight.
MAKE A YEAR END DONATION—HELP EFF UNLOCK CHALLENGE GRANTS!
If you donate to EFF before the end of 2025, you’ll help fuel the legal battles that defend encryption, the tools that protect privacy, and the advocacy that stops dangerous laws—and you’ll help unlock up to $26,200 in challenge grants.
The past year confirmed how urgently we need technologies that protect us, not surveil us. EFF has been in the fight every step of the way, thanks to support from people like you.
Get free gear when you join EFF!
This year alone EFF:
After 35 years of defending digital freedoms, we know what's at stake: we must protect your ability to speak freely, organize safely, and use technology without surveillance.
We have opportunities to win these fights, and you make each victory possible. Donate to EFF by December 31 and help us unlock additional grants this year!
EFF Members have carried the movement for privacy and free expression for decades. You can help move the mission even further! Here’s some sample language that you can share with your networks:
We need to stand together and ensure technology works for us, not against us. Donate any amount to EFF by Dec 31, and you'll help unlock challenge grants! https://eff.org/yec
Bluesky | Facebook | LinkedIn | Mastodon
(more at eff.org/social)
_________________
EFF is a member-supported U.S. 501(c)(3) organization. We’re celebrating TWELVE YEARS of top ratings from the nonprofit watchdog Charity Navigator! Your donation is tax-deductible as allowed by law.
Another year has come and gone, and with it, thousands of data breaches that affect millions of people. The question these days is less, Is my information in a data breach this year? and more How many data breaches had my information in them this year?
Some data breaches are more noteworthy than others. Where one might affect a small number of people and include little useful information, like a name or email address, others might include data ranging from a potential medical diagnosis to specific location information. To catalog and talk about these breaches we created the Breachies, a series of tongue-in-cheek awards, to highlight the most egregious data breaches.
In most cases, if these companies practiced a privacy first approach and focused on data minimization, only collecting and storing what they absolutely need to provide the services they promise, many data breaches would be far less harmful to the victims. But instead, companies gobble up as much as they can, store it for as long as possible, and inevitably at some point someone decides to poke in and steal that data. Once all that personal data is stolen, it can be used against the breach victims for identity theft, ransomware attacks, and to send unwanted spam. It has become such a common occurrence that it’s easy to lose track of which breaches affect you, and just assume your information is out there somewhere. Still, a few steps can help protect your information.
With that, let’s get to the awards.
The Winners
We’ve long warned that apps delivering your personal information to third-parties, even if they aren’t the ad networks directly driving surveillance capitalism, presents risks and a salient target for hackers. The more widespread your data, the more places attackers can go to find it. Mixpanel, a data analytics company which collects information on users of any app which incorporates its SDK, suffered a major breach in November this year. The service has been used by a wide array of companies, including the Ring Doorbell App, which we reported on back in 2020 delivering a trove of information to Mixpanel, and PornHub, which despite not having worked with the company since 2021, had its historical record of paying subscribers breached.
There’s a lot we still don’t know about this data breach, in large part because the announcement about it is so opaque, leaving reporters with unanswered questions about how many were affected, if the hackers demanded a ransom, and if Mixpanel employee accounts utilized standard security best practices. One thing is clear, though: the breach was enough for OpenAI to drop them as a provider, disclosing critical details on the breach in a blog post that Mixpanel’s own announcement conveniently failed to mention.
The worst part is that, as a data analytics company providing libraries which are included in a broad range of apps, we can surmise that the vast majority of people affected by this breach have no direct relationship with Mixpanel, and likely didn’t even know that their devices were delivering data to the company. These people deserve better than vague statements by companies which profit off of (and apparently insufficiently secure) their data.
Last year, AU10TIX won our first The We Told You So Award because as we predicted in 2023, age verification mandates would inevitably lead to more data breaches, potentially exposing government IDs as well as information about the sites that a user visits. Like clockwork, they did. It was our first We Told You So Breachies award, but we knew it wouldn’t be the last.
Unfortunately, there is growing political interest in mandating identity or age verification before allowing people to access social media or adult material. EFF and others oppose these plans because they threaten both speech and privacy.
Nonetheless, this year’s winner of The We Still Told You So Breachies Award is the messaging app, Discord — once known mainly for gaming communities, it now hosts more than 200 million monthly active users and is widely used to host fandom and community channels.
In September of this year, much of Discord’s age verification data was breached — including users’ real names, selfies, ID documents, email and physical addresses, phone numbers, IP addresses, and other contact details or messages provided to customer support. In some cases, “limited billing information” was also accessed—including payment type, the last four digits of credit card numbers, and purchase histories.
Technically though, it wasn’t Discord itself that was hacked but their third-party customer support provider — a company called Zendesk—that was compromised, allowing attackers to access Discord’s user data. Either way, it’s Discord users who felt the impact.
Speaking of age verification, Tea, the dating safety app for women, had a pretty horrible year for data breaches. The app allows users to anonymously share reviews and safety information about their dates with men—helping keep others safe by noting red flags they saw during their date.
Since Tea is aimed at women’s safety and dating advice, the app asks new users to upload a selfie or photo ID to verify their identity and gender to create an account. That’s some pretty sensitive information that the app is asking you to trust it with! Back in July, it was reported that 72,000 images had been leaked from the app, including 13,000 images of photo IDs and 59,000 selfies. These photos were found via an exposed database hosted on Google’s mobile app development platform, Firebase. And if that isn’t bad enough, just a week later a second breach exposed private messages between users, including messages with phone numbers, abortion planning, and discussions about cheating partners. This breach included more than 1.1 million messages from early 2023 all the way to mid-2025, just before the breach was reported. Tea released a statement shortly after, temporarily disabling the chat feature.
But wait, there’s more. A completely different app based on the same idea, but for men, also suffered a data breach. TeaOnHer failed to protect similar sensitive data. In August, TechCrunch discovered that user information — including emails, usernames, and yes, those photo IDs and selfies — was accessible through a publicly available web address. Even worse? TechCrunch also found the email address and password the app’s creator uses to access the admin page.
Breaches like this are one of the reasons that EFF shouts from the rooftops against laws that mandate user verification with an ID or selfie. Every company that collects this information becomes a target for data breaches — and if a breach happens, you can’t just change your face.
Another year, another data breach caused by online tracking tools.
In April, Blue Shield of California revealed that it had shared 4.7 million people’s health data with Google by misconfiguring Google Analytics on its website. The data, which may have been used for targeted advertising, included: people’s names, insurance plan details, medical service providers, and patient financial responsibility. The health insurance company shared this information with Google for nearly three years before realizing its mistake.
If this data breach sounds familiar, it’s because it is: last year’s Just Stop Using Tracking Tech award also went to a healthcare company that leaked patient data through tracking code on its website. Tracking tools remain alarmingly common on healthcare websites, even after years of incidents like this one. These tools are marketed as harmless analytics or marketing solutions, but can expose people’s sensitive data to advertisers and data brokers.
EFF’s free Privacy Badger extension can block online trackers, but you shouldn’t need an extension to stop companies from harvesting and monetizing your medical data. We need a strong, federal privacy law and ban on online behavioral advertising to eliminate the incentives driving companies to keep surveilling us online.
In December 2024, PowerSchool, the largest provider of student information systems in the U.S., gave hackers access to sensitive student data. The breach compromised personal information of over 60 million students and teachers, including Social Security numbers, medical records, grades, and special education data. Hackers exploited PowerSchool’s weak security—namely, stolen credentials to their internal customer support portal—and gained unfettered access to sensitive data stored by school districts across the country.
PowerSchool failed to implement basic security measures like multi-factor authentication, and the breach affected districts nationwide. In Texas alone, over 880,000 individuals’ data was exposed, prompting the state's attorney general to file a lawsuit, accusing PowerSchool of misleading its customers about security practices. Memphis-Shelby County Schools also filed suit, seeking damages for the breach and the cost of recovery.
While PowerSchool paid hackers an undisclosed sum to prevent data from being published, the company’s failure to protect its users’ data raises serious concerns about the security of K-12 educational systems. Adding to the saga, a Massachusetts student, Matthew Lane, pleaded guilty in October to hacking and extorting PowerSchool for $2.85 million in Bitcoin. Lane faces up to 17 years in prison for cyber extortion and aggravated identity theft, a reminder that not all hackers are faceless shadowy figures — sometimes they’re just a college kid.
Credit reporting giant TransUnion had to notify its customers this year that a hack nabbed the personal information of 4.4 million people. How'd the attackers get in? According to a letter filed with the Maine Attorney General's office obtained by TechCrunch, the problem was a “third-party application serving our U.S. consumer support operations.” That's probably not the kind of support they were looking for.
TransUnion said in a Texas filing that attackers swept up “customers’ names, dates of birth, and Social Security numbers” in the breach, though it was quick to point out in public statements that the hackers did not access credit reports or “core credit data.” While it certainly could have been worse, this breach highlights the many ways that hackers can get their hands on information. Coming in through third-parties, companies that provide software or other services to businesses, is like using an unguarded side door, rather than checking in at the front desk. Companies, particularly those who keep sensitive personal information, should be sure to lock down customer information at all the entry points. After all, their decisions about who they do business with ultimately carry consequences for all of their customers — who have no say in the matter.
Microsoft is a company nobody feels neutral about. Especially in the infosec world. The myriad software vulnerabilities in Windows, Office, and other Microsoft products over the decades has been a source of frustration and also great financial rewards for both attackers and defenders. Yet still, as the saying goes: “nobody ever got fired for buying from Microsoft.” But perhaps, the times, they are a-changing.
In July 2025, it was revealed that a zero-day security vulnerability in Microsoft’s flagship file sharing and collaboration software, SharePoint, had led to the compromise of over 400 organizations, including major corporations and sensitive government agencies such as the National Nuclear Security Administration (NNSA), the federal agency responsible for maintaining and developing the U.S. stockpile of nuclear weapons. The attack was attributed to three different Chinese government linked hacking groups. Amazingly, days after the vulnerability was first reported, there were still thousands of vulnerable self-hosted Sharepoint servers online.
Zero-days happen to tech companies, large and small. It’s nearly impossible to write even moderately complex software that is bug and exploit free, and Microsoft can’t exactly be blamed for having a zero-day in their code. But when one company is the source of so many zero-days consistently for so many years, one must start wondering whether they should put all their eggs (or data) into a basket that company made. Perhaps if Microsoft’s monopolistic practices had been reined in back in the 1990s we wouldn’t be in a position today where Sharepoint is the defacto file sharing software for so many major organizations. And maybe, just maybe, this is further evidence that tech monopolies and centralization of data aren’t just bad for consumer rights, civil liberties, and the economy—but also for cybersecurity.
Look, we’ll keep this one short: in October of last year, researchers found security issues in the flat earther app, Flat Earth, Sun, Moon, & Clock. In March of 2025, that breach was confirmed. What’s most notable about this, aside from including a surprising amount of information about gender, name, email addresses and date of birth, is that it also included users’ location info, including latitude and longitude. Huh, interesting.
In January, hackers claimed they stole millions of people’s location history from a company that never should’ve had it in the first place: location data broker Gravy Analytics. The data included timestamped location coordinates tied to advertising IDs, which can reveal exceptionally sensitive information. In fact, researchers who reviewed the leaked data found it could be used to identify military personnel and gay people in countries where homosexuality is illegal.
The breach of this sensitive data is bad, but Gravy Analytics’s business model of regularly harvesting and selling it is even worse. Despite the fact that most people have never heard of them, Gravy Analytics has managed to collect location information from a billion phones a day. The company has sold this data to other data brokers, makers of police surveillance tools, and the U.S. government.
How did Gravy Analytics get this location information from people’s phones? The data broker industry is notoriously opaque, but this breach may have revealed some of Gravy Analytics’ sources. The leaked data referenced thousands of apps, including Microsoft apps, Candy Crush, Tinder, Grindr, MyFitnessPal, pregnancy trackers and religious-focused apps. Many of these app developers said they had no relationship with Gravy Analytics. Instead, expert analysis of the data suggests it was harvested through the advertising ecosystem already connected to most apps. This breach provides further evidence that online behavioral advertising fuels the surveillance industry.
Whether or not they get hacked, location data brokers like Gravy Analytics threaten our privacy and security. Follow EFF’s guide to protecting your location data and help us fight for legislation to dismantle the data broker industry.
TeslaMate, a tool meant to track Tesla vehicle data (but which is not owned or operated by Tesla itself), has become a cautionary tale about data security. In August, a security researcher found more than 1,300 self-hosted TeslaMate dashboards were exposed online, leaking sensitive information such as vehicle location, speed, charging habits, and even trip details. In essence, your Cybertruck became the star of its own Keeping Up With My Cybertruck reality show, except the audience wasn’t made up of fans interested in your lifestyle, just random people with access to the internet.
TeslaMate describes itself as “that loyal friend who never forgets anything!” — but its lack of proper security measures makes you wish it would. This breach highlights how easily location data can become a tool for harassment or worse, and the growing need for legislation that specifically protects consumer location data. Without stronger regulations around data privacy, sensitive location details like where you live, work, and travel can easily be accessed by malicious actors, leaving consumers with no recourse.
Confidentiality is a core principle in the practice of law. But this year a breach of confidentiality came from an unexpected source: a breach of the federal court filing system. In August, Politico reported that hackers infiltrated the Case Management/Electronic Case Files (CM/ECF) system, which uses the same database as PACER, a searchable public database for court records. Of particular concern? The possibility that the attack exposed the names of confidential informants involved in federal cases from multiple court districts. Courts across the country acted quickly to set up new processes to avoid the possibility of further compromises.
The leak followed a similar incident in 2021 and came on the heels of a warning to Congress that the file system is more than a little creaky. In fact, an IT official from the federal court system told the House Judiciary Committee that both systems are “unsustainable due to cyber risks, and require replacement.”
Just like last year, a stalkerware company was subject to a data breach that really should prove once and for all that these companies must be stopped. In this case, Catwatchful is an Android spyware company that sells itself as a “child monitoring app.” Like other products in this category, it’s designed to operate covertly while uploading the contents of a victim’s phone, including photos, messages, and location information.
This data breach was particularly harmful, as it included not just the email addresses and passwords on the customers who purchased the app to install on a victim’s phone, but also the data from the phones of 26,000 victims’ devices, which could include the victims’ photos, messages, and real-time location data.
This was a tough award to decide on because Catwatchful wasn’t the only stalkerware company that was hit this year. Similar breaches to SpyX, Cocospy, and Spyic were all strong contenders. EFF has worked tirelessly to raise the alarm on this sort of software, and this year worked with AV Comparatives to test the stalkerware detection rate on Android of various major antivirus apps.
Every year, we all get a reminder about why using unique passwords for all our accounts is crucial for protecting our online identities. This time around, the award goes to Plex, who experienced a data breach that included customer emails, usernames, and hashed passwords (which is a fancy way of saying passwords are scrambled through an algorithm, but it is possible they could still be deciphered).
If this all sounds vaguely familiar to you for some reason, that’s because a similar issue also happened to Plex in 2022, affecting 15 million users. Whoops.
This is why it is important to use unique passwords everywhere. A password manager, including one that might be free on your phone or browser, makes this much easier to do. Likewise, credential stuffing illustrates why it’s important to use two-factor authentication. Here’s how to turn that on for your Plex account.
Troy Hunt, the person behind Have I Been Pwned? and who has more experience with data breaches than just about anyone, also proved that anyone can be pwned. In a blog post, he details what happened to his mailing list:
You know when you're really jet lagged and really tired and the cogs in your head are just moving that little bit too slow? That's me right now, and the penny has just dropped that a Mailchimp phish has grabbed my credentials, logged into my account and exported the mailing list for this blog.
And he continues later:
I'm enormously frustrated with myself for having fallen for this, and I apologise to anyone on that list. Obviously, watch out for spam or further phishes and check back here or via the social channels in the nav bar above for more.
The whole blog is worth a read as a reminder that phishing can get anyone, and we thank Troy Hunt for his feedback on this and other breaches to include this year.
Data breaches are such a common occurrence that it’s easy to feel like there’s nothing you can do, nor any point in trying. But privacy isn’t dead. While some information about you is almost certainly out there, that’s no reason for despair. In fact, it’s a good reason to take action.
There are steps you can take right now with all your online accounts to best protect yourself from the the next data breach (and the next, and the next):
According to one report, 2025 had already seen 2,563 data breaches by October, which puts the year on track to be one of the worst by the sheer number of breaches.
We did not investigate every one of these 2,500-plus data breaches, but we looked at a lot of them, including the news coverage and the data breach notification letters that many state Attorney General offices host on their websites. We can’t award the coveted Breachies Award to every company that was breached this year. Still, here are some (dis)honorable mentions we wanted to highlight:
Salesforce, F5, Oracle, WorkComposer, Raw, Stiizy, Ohio Medical Alliance LLC, Hello Cake, Lovense, Kettering Health, LexisNexis, WhatsApp, Nexar, McDonalds, Congressional Budget Office, Doordash, Louis Vuitton, Adidas, Columbia University, Hertz, HCRG Care Group, Lexipol, Color Dating, Workday, Aflac, and Coinbase. And a special nod to last minute entrants Home Depot, 700Credit, and Petco.
What now? Companies need to do a better job of only collecting the information they need to operate, and properly securing what they store. Also, the U.S. needs to pass comprehensive privacy protections. At the very least, we need to be able to sue companies when these sorts of breaches happen (and while we’re at it, it’d be nice if we got more than $5.21 checks in the mail). EFF has long advocated for a strong federal privacy law that includes a private right of action.
The final EFFector of 2025 is here! Just in time to keep you up-to-date on the latests happenings in the fight for privacy and free speech online.
In this latest issue, we're sharing how to spot sneaky ALPR cameras at the U.S. border, covering a host of new resources on age verification laws, and explaining why AI companies need to protect chatbot logs from bulk surveillance.
Prefer to listen in? Check out our audio companion, where EFF Activist Molly Buckley explains our new resource explaining age verification laws and how you can fight back. Catch the conversation on YouTube or the Internet Archive.
EFFECTOR 37.18 - 🪪 AGE VERIFICATION IS COMING FOR THE INTERNET
Since 1990 EFF has published EFFector to help keep readers on the bleeding edge of their digital rights. We know that the intersection of technology, civil liberties, human rights, and the law can be complicated, so EFFector is a great way to stay on top of things. The newsletter is chock full of links to updates, announcements, blog posts, and other stories to help keep readers—and listeners—up to date on the movement to protect online privacy and free expression.
Thank you to the supporters around the world who make our work possible! If you're not a member yet, join EFF today to help us fight for a brighter digital future.
If you’re a Californian, there are a few new state laws that you should know will be going into effect in the new year. EFF has worked hard in Sacramento this session to advance bills that protect privacy, fight surveillance, and promote transparency.
California’s legislature runs in a two-year cycle, meaning that it’s currently halftime for legislators. As we prepare for the next year of the California legislative session in January, it’s a good time to showcase what’s happened so far—and what’s left to do.
In a win for every Californian’s privacy rights, we were happy to support A.B. 566 (Assemblymember Josh Lowenthal). This is a common-sense law that makes California’s main consumer data privacy law, the California Consumer Privacy Act, more user-friendly. It requires that browsers support people’s rights to send opt-out signals, such as the global opt-out in Privacy Badger, to businesses. Managing your privacy as an individual can be a hard job, and EFF wants stronger laws that make it easier for you to do so.
Additionally, we were proud to advance government transparency by supporting A.B. 1524 (Judiciary Committee), which allows members of the public to make copies of public court records using their own devices, such as cell-phone cameras and overhead document scanners, without paying fees.
We also supported two bills that will improve law enforcement accountability at a time when we desperately need it. S.B. 627 (Senator Scott Wiener) prohibits law enforcement officers from wearing masks to avoid accountability (The Trump administration has sued California over this law). We also supported S.B. 524 (Asm. Jesse Arreguín), which requires law enforcement to disclose when a police report was written using artificial intelligence.
On the flip side, we also stopped some problematic bills from becoming law. This includes S.B. 690 (Sen. Anna Caballero), which we dubbed the Corporate Coverup Act. This bill would have gutted California’s wiretapping statute by allowing businesses to ignore those privacy rights for “any business purpose.” Working with several coalition partners, we were able to keep that bill from moving forward in 2025. We do expect to see it come back in 2026, and are ready to fight back against those corporate business interests.
And, of course, not every fight ended in victory. There are still many areas where we have work left to do. California Governor Gavin Newsom vetoed a bill we supported, S.B. 7, which would have given workers in California greater transparency into how their employers use artificial intelligence and was sponsored by the California Federation of Labor Unions. S.B. 7 was vetoed in response to concerns from companies including Uber and Lyft, but we expect to continue working with the labor community on the ways AI affects the workplace in 2026.
California continued a troubling years-long trend of lawmakers pushing problematic proposals that would require every internet user to verify their age to access information—often by relying on privacy-invasive methods to do so. Earlier this year EFF sent a letter to the California legislature expressing grave concerns with lawmakers’ approach to regulating young people’s ability to speak online. We continue to raise these concerns, and would welcome working with any lawmaker in California on a better solution.
We also continue to keep a close eye on government data sharing. On this front, there is some good news. Several of the bills we supported this year sought to place needed safeguards on the ways various government agencies in California share data. These include: A.B. 82 (Asm. Chris Ward) and S.B. 497 (Wiener), which would add privacy protections to data collected by the state about those who may be receiving gender-affirming or reproductive health care; A.B. 1303 (Asm. Avelino Valencia), which prohibits warrantless data sharing from California’s low-income broadband program to immigration and other government officials; and S.B. 635 (Sen. Maria Elena Durazo), which places similar limits on data collected from sidewalk vendors.
We are also heartened to see California correct course on broad government data sharing. Last session, we opposed A.B. 518 (Asm. Buffy Wicks), which let state agencies ignore existing state privacy law to allow broader information sharing about people eligible for CalFresh—the state’s federally funded food assistance program. As we’ve seen, the federal government has since sought data from food assistance programs to use for other purposes. We were happy to have instead supported A.B. 593 this year, also authored by Asm. Wicks—which reversed course on that data sharing.
We hope to see this attention to the harms of careless government data sharing continue. EFF’s sponsored bill this year, A.B. 1337, would update and extend vital privacy safeguards present at the state agency level to counties and cities. These local entities today collect enormous amounts of data and administer programs that weren’t contemplated when the original law was written in 1977. That information should be held to strong privacy standards.
We’ve been fortunate to work with Asm. Chris Ward, who is also the chair of the LGBTQ Caucus in the legislature, on that bill. The bill stalled in the Senate Judiciary Committee during the 2025 legislative session, but we plan to bring it back in the next session with a renewed sense of urgency.
Since the Online Safety Act took effect in late July, UK internet users have made it very clear to their politicians that they do not want anything to do with this censorship regime. Just days after age checks came into effect, VPN apps became the most downloaded on Apple's App Store in the UK, and a petition calling for the repeal of the Online Safety Act (OSA) hit over 400,000 signatures.
In the months since, more than 550,000 people have petitioned Parliament to repeal or reform the Online Safety Act, making it one of the largest public expressions of concern about a UK digital law in recent history. The OSA has galvanized swathes of the UK population, and it’s high time for politicians to take that seriously.
Last week, EFF joined Open Rights Group, Big Brother Watch, and Index on Censorship in sending a briefing to UK politicians urging them to listen to their constituents and reform or repeal the Online Safety Act ahead of this week’s Parliamentary petition debate on 15 December.
The legislation is a threat to user privacy, restricts free expression by arbitrating speech online, exposes users to algorithmic discrimination through face checks, and effectively blocks millions of people without a personal device or form of ID from accessing the internet. The briefing highlights how, in the months since the OSA came into effect, we have seen the legislation:
Our briefing continues:
“Those raising concerns about the Online Safety Act are not opposing child safety. They are asking for a law that does both: protects children and respects fundamental rights, including children’s own freedom of expression rights.”
The petition shows that hundreds of thousands of people feel the current Act tilts too far, creating unnecessary risks for free expression and ordinary online life. With sensible adjustments, Parliament can restore confidence that online safety and freedom of expression rights can coexist.
If the UK really wants to achieve its goal of being the safest place in the world to go online, it must lead the way in introducing policies that actually protect all users—including children—rather than pushing the enforcement of legislation that harms the very people it was meant to protect.
Read the briefing in full here.
Update, 17 Dec 2025: this article was edited to include the word reform alongside repeal.
The UK Parliament convened earlier this week to debate a petition signed by 2.9 million people calling for an end to the government’s plans to roll out a national digital ID. Ahead of that debate, EFF and 12 other civil society organizations wrote to politicians in the country urging MPs to reject the Labour government’s newly announced digital ID proposal.
The UK’s Prime Minister Keir Starmer pitched the scheme as a way to “cut the faff” in proving people’s identities by creating a virtual ID on personal devices with information like names, date of birth, nationality, photo, and residency status to verify their right to live and work in the country.
But the case for digital identification has not been made.
As we detail in our joint briefing, the proposal follows a troubling global trend: governments introducing expansive digital identity systems that are structurally incompatible with a rights-respecting democracy. The UK’s plan raises six interconnected concerns:
Digital ID schemes don’t simply verify who you are—they redefine who can access services and what those services look like. They become a gatekeeper to essential societal infrastructure, enabling governments and state agencies to close doors as easily as they open them. And they disproportionately harm those already at society’s margins, including people seeking asylum and undocumented communities, who already face heightened surveillance and risk.
Even the strongest recommended safeguards cannot resolve the core problem: a mandatory digital ID scheme that shifts power dramatically away from individuals and toward the state. No one should be coerced—technically or socially—into a digital system in order to participate fully in public life. And at a time when almost 3 million people in the UK have called on politicians to reject this proposal, the government must listen to people and say no to digital ID.
Read our civil society briefing in full here.
A massive wave of public comments just told the U.S. Patent and Trademark Office (USPTO): don’t shut the public out of patent review.
EFF submitted its own formal comment opposing the USPTO’s proposed rules, and more than 4,000 supporters added their voices—an extraordinary response for a technical, fast-moving rulemaking. We comprised more than one-third of the 11,442 comments submitted. The message is unmistakable: the public wants a meaningful way to challenge bad patents, and the USPTO should not take that away.
These thousands of submissions do more than express frustration. They demonstrate overwhelming public interest in preserving inter partes review (IPR), and undermine any broad claim that the USPTO’s proposal reflects public sentiment.
Comments opposing the rulemaking include many small business owners who have been wrongly accused of patent infringement, by both patent trolls and patent-abusing competitors. They also include computer science experts, law professors, and everyday technology users who are simply tired of patent extortion—abusive assertions of low-quality patents—and the harm it inflicts on their work, their lives, and the broader U.S. economy.
The USPTO exists to serve the public. The volume and clarity of this response make that expectation impossible to ignore.
In our filing, we explained that the proposed rules would make it significantly harder for the public to challenge weak patents. That undercuts the very purpose of IPR. The proposed rules would pressure defendants to give up core legal defenses, allow early or incomplete decisions to block all future challenges, and create new opportunities for patent owners to game timing and shut down PTAB review entirely.
Congress created IPR to allow the Patent Office to correct its own mistakes in a fair, fast, expert forum. These changes would take the system backward.
A wide range of groups told the USPTO the same thing: don’t cut off access to IPR.
Open Source and Developer Communities
The Linux Foundation submitted comments and warned that the proposed rules “would effectively remove IPRs as a viable mechanism for challenges to patent validity,” harming open-source developers and the users that rely on them. Github wrote that the USPTO proposal would increase “litigation risk and costs for developers, startups, and open source projects.” And dozens of individual software developers described how bad patents have burdened their work.
Patent Law Scholars
A group of 22 patent law professors from universities across the country said the proposed rule changes “would violate the law, increase the cost of innovation, and harm the quality of patents.”
Patient Advocates
Patients for Affordable Drugs warned in their filing that IPR is critical for invalidating wrongly granted pharmaceutical patents. When such patents are invalidated, studies have shown “cardiovascular medications have fallen 97% in price, cancer drugs dropping 80-98%, and treatments for opioid addiction becom[e] 50% more affordable.” In addition, “these cases involved patents that had evaded meaningful scrutiny in district court.”
Small Businesses
Hundreds of small businesses weighed in with a consistent message: these proposed rules would hit them hardest. Owners and engineers described being targeted with vague or overbroad patents they cannot afford to litigate in court, explaining that IPR is often the only realistic way for a small firm to defend itself. The proposed rules would leave them with an impossible choice—pay a patent troll, or spend money they don’t have fighting in federal court.
The USPTO now has thousands of comments to review. It should listen. Public participation must be more than a box-checking exercise. It is central to how administrative rulemaking is supposed to work.
Congress created IPR so the public could help correct bad patents without spending millions of dollars in federal court. People across technical, academic, and patient-advocacy communities just reminded the agency why that matters.
We hope the USPTO reconsiders these proposed rules. Whatever happens, EFF will remain engaged and continue fighting to preserve the public’s ability to challenge bad patents.
This blog also appears in our Age Verification Resource Hub: our one-stop shop for users seeking to understand what age-gating laws actually do, what’s at stake, how to protect yourself, and why EFF opposes all forms of age verification mandates. Head to EFF.org/Age to explore our resources and join us in the fight for a free, open, private, and yes—safe—internet.
One of the most common refrains we hear from age verification proponents is that online ID checks are nothing new. After all, you show your ID at bars and liquor stores all the time, right? And it’s true that many places age-restrict access in-person to various goods and services, such as tobacco, alcohol, firearms, lottery tickets, and even tattoos and body piercings.
But this comparison falls apart under scrutiny. There are fundamental differences between flashing your ID to a bartender and uploading government documents or biometric data to websites and third-party verification companies. Online age-gating is more invasive, affects far more people, and poses serious risks to privacy, security, and free speech that simply don't exist when you buy a six-pack at the corner store.
Online age restrictions are imposed on many, many more users than in-person ID checks. Because of the sheer scale of the internet, regulations affecting online content sweep in an enormous number of adults and youth alike, forcing them to disclose sensitive personal data just to access lawful speech, information, and services.
Additionally, age restrictions in the physical world affect only a limited number of transactions: those involving a narrow set of age-restricted products or services. Typically this entails a bounded interaction about one specific purchase.
Online age verification laws, on the other hand, target a broad range of internet activities and general purpose platforms and services, including social media sites and app stores. And these laws don’t just wall off specific content deemed harmful to minors (like a bookstore would); they age-gate access to websites wholesale. This is akin to requiring ID every time a customer walks into a convenience store, regardless of whether they want to buy candy or alcohol.
In offline, in-person scenarios, a customer typically provides their physical ID to a cashier or clerk directly. Oftentimes, customers need only flash their ID for a quick visual check, and no personal information is uploaded to the internet, transferred to a third-party vendor, or stored. Online age-gating, on the other hand, forces users to upload—not just momentarily display—sensitive personal information to a website in order to gain access to age-restricted content.
This creates a cascade of privacy and security problems that don’t exist in the physical world. Once sensitive information like a government-issued ID is uploaded to a website or third-party service, there is no guarantee it will be handled securely. You have no direct control over who receives and stores your personal data, where it is sent, or how it may be accessed, used, or leaked outside the immediate verification process.
Data submitted online rarely just stays between you and one other party. All online data is transmitted through a host of third-party intermediaries, and almost all websites and services also host a network of dozens of private, third-party trackers managed by data brokers, advertisers, and other companies that are constantly collecting data about your browsing activity. The data is shared with or sold to additional third parties and used to target behavioral advertisements. Age verification tools also often rely on third parties just to complete a transaction: a single instance of ID verification might involve two or three different third-party partners, and age estimation services often work directly with data brokers to offer a complete product. Users’ personal identifying data then circulates among these partners.
All of this increases the likelihood that your data will leak or be misused. Unfortunately, data breaches are an endemic part of modern life, and the sensitive, often immutable, personal data required for age verification is just as susceptible to being breached as any other online data. Age verification companies can be—and already have been—hacked. Once that personal data gets into the wrong hands, victims are vulnerable to targeted attacks both online and off, including fraud and identity theft.
Troublingly, many age verification laws don’t even protect user security by providing a private right of action to sue a company if personal data is breached or misused. This leaves you without a direct remedy should something bad happen.
Some proponents claim that age estimation is a privacy-preserving alternative to ID-based verification. But age estimation tools still require biometric data collection, often demanding users submit a photo or video of their face to access a site. And again, once submitted, there’s no way for you to verify how that data is processed or stored. Requiring face scans also normalizes pervasive biometric surveillance and creates infrastructure that could easily be repurposed for more invasive tracking. Once we’ve accepted that accessing lawful speech requires submitting our faces for scanning, we’ve crossed a threshold that’s difficult to walk back.
Online age gates create more substantial access barriers than in-person ID checks do. For those concerned about privacy and security, there is no online analog to a quick visual check of your physical ID. Users may be justifiably discouraged from accessing age-gated websites if doing so means uploading personal data and creating a potentially lasting record of their visit to that site.
Given these risks, age verification also imposes barriers to remaining anonymous that don't typically exist in-person. Anonymity can be essential for those wishing to access sensitive, personal, or stigmatized content online. And users have a right to anonymity, which is “an aspect of the freedom of speech protected by the First Amendment.” Even if a law requires data deletion, users must still be confident that every website and online service with access to their data will, in fact, delete it—something that is in no way guaranteed.
In-person ID checks are additionally less likely to wrongfully exclude people due to errors. Online systems that rely on facial scans are often incorrect, especially when applied to users near the legal age of adulthood. These tools are also less accurate for people with Black, Asian, Indigenous, and Southeast Asian backgrounds, for users with disabilities, and for transgender individuals. This leads to discriminatory outcomes and exacerbates harm to already marginalized communities. And while in-person shoppers can speak with a store clerk if issues arise, these online systems often rely on AI models, leaving users who are incorrectly flagged as minors with little recourse to challenge the decision.
In-person interactions may also be less burdensome for adults who don’t have up-to-date ID. An older adult who forgets their ID at home or lacks current identification is not likely to face the same difficulty accessing material in a physical store, since there are usually distinguishing physical differences between young adults and those older than 35. A visual check is often enough. This matters, as a significant portion of the U.S. population does not have access to up-to-date government-issued IDs. This disproportionately affects Black Americans, Hispanic Americans, immigrants, and individuals with disabilities, who are less likely to possess the necessary identification.
It's important not to lose sight of what’s at stake here. The good or service age gated by these laws isn’t alcohol or cigarettes—it’s First Amendment-protected speech. Whether the target is social media platforms or any other online forum for expression, age verification blocks access to constitutionally-protected content.
Access to many of these online services is also necessary to participate in the modern economy. While those without ID may function just fine without being able to purchase luxury products like alcohol or tobacco, requiring ID to participate in basic communication technology significantly hinders people’s ability to engage in economic and social life.
This is why it’s wrong to claim online age verification is equivalent to showing ID at a bar or store. This argument handwaves away genuine harms to privacy and security, dismisses barriers to access that will lock millions out of online spaces, and ignores how these systems threaten free expression. Ignoring these threats won’t protect children, but it will compromise our rights and safety.
Age verification laws are proliferating fast across the United States and around the world, creating a dangerous and confusing tangle of rules about what we’re all allowed to see and do online. Though these mandates claim to protect children, in practice they create harmful censorship and surveillance regimes that put everyone—adults and young people alike—at risk.
The term “age verification” is colloquially used to describe a wide range of age assurance technologies, from age verification systems that force you to upload government ID, to age estimation tools that scan your face, to systems that infer your age by making you share personal data. While different laws call for different methods, one thing remains constant: every method out there collects your sensitive, personal information and creates barriers to accessing the internet. We refer to all of these requirements as age verification, age assurance, or age-gating.
If you’re feeling overwhelmed by this onslaught of laws and the invasive technologies behind them, you’re not alone. It’s a lot. But understanding how these mandates work and who they harm is critical to keeping yourself and your loved ones safe online. Age verification is lurking around every corner these days, so we must fight back to protect the internet that we know and love.
That’s why today, we’re launching EFF’s Age Verification Resource Hub (EFF.org/Age): a one-stop shop to understand what these laws actually do, what’s at stake, why EFF opposes all forms of age verification, how to protect yourself, and how to join the fight for a free, open, private, and yes—safe—internet.
In the U.S., more than half of all states have now passed laws imposing age-verification requirements on online platforms. Congress is considering even more at the federal level, with a recent House hearing weighing nineteen distinct proposals relating to young people’s online safety—some sweeping, some contradictory, and each one more drastic and draconian than the last.
We all want young people to be safe online. However, age verification is not the silver bullet that lawmakers want you to think it is.
The rest of the world is moving in the same direction. We saw the UK’s Online Safety Act go into effect this summer, Australia’s new law barring access to social media for anyone under 16 goes live today, and a slew of other countries are currently considering similar restrictions.
We all want young people to be safe online. However, age verification is not the silver bullet that lawmakers want you to think it is. In fact, age-gating mandates will do more harm than good—especially for the young people they claim to protect. They undermine the fundamental speech rights of adults and young people alike; create new barriers to accessing vibrant, lawful, even life-saving content; and needlessly jeopardize all internet users’ privacy, anonymity, and security.
If legislators want to meaningfully improve online safety, they should pass a strong, comprehensive federal privacy law instead of building new systems of surveillance, censorship, and exclusion.
Our new hub is built to answer the questions we hear from users every day, such as:
Head over to EFF.org/Age to explore our explainers, user-friendly guides, technical breakdowns, and advocacy tools—all indexed in the sidebar for easy browsing. And today is just the start, so keep checking back over the next several weeks as we continue to build out the site with new resources and answers to more of your questions on all things age verification.
To celebrate the launch of EFF.org/Age, and to hear directly from you how we can be most helpful in this fight, we’re hosting two exciting events:
Next week, our team of EFF activists, technologists, and lawyers will be hanging out over on Reddit’s r/privacy subreddit to directly answer your questions on all things age verification. We’re looking forward to connecting with you and hearing how we can help you navigate these changing tides, so come on over to r/privacy on Monday (12/15), Tuesday (12/16), and Wednesday (12/17), and ask us anything!
Then, on January 15th at 12pm PT, we’re hosting a livestream panel featuring Cynthia Conti-Cook, Director of Research and Policy at the Collaborative Research Center for Resilience; Hana Memon, Software Developer at Gen Z for Change; EFF Director of Engineering Alexis Hancock; and EFF Associate Director of State Affairs Rindala Alajaji. We’ll break down how these laws work, who they exclude, and how these mandates threaten privacy and free expression for people of all ages. Join us by RSVPing at https://livestream.eff.org/.
Age-verification mandates are reshaping the internet in ways that are invasive, dangerous, and deeply unnecessary. But users are not powerless! We can challenge these laws, protect our digital rights, and build a safer digital world for all internet users, no matter their ages. Our new resource hub is here to help—so explore, share, and join us in the fight for a better internet.
The state of streaming is... bad. It’s very bad. The first step in wanting to watch anything is a web search: “Where can I stream X?” Then you have to scroll past an AI summary with no answers, and then scroll past the sponsored links. After that, you find out that the thing you want to watch was made by a studio that doesn’t exist anymore or doesn’t have a streaming service. So, even though you subscribe to more streaming services than you could actually name, you will have to buy a digital copy to watch. A copy that, despite paying for it specifically, you do not actually own and might vanish in a few years.
Then, after you paid to see something multiple times in multiple ways (theater ticket, VHS tape, DVD, etc.), the mega-corporations behind this nightmare will try to get Congress to pass laws to ensure you keep paying them. In the end, this is easier than making a product that works. Or, as someone put it on social media, these companies have forgotten “that their entire existence relies on being slightly more convenient than piracy.”
It’s important to recognize this as we see more and more media mergers. These mergers are not about quality, they’re about control.
In the old days, studios made a TV show. If the show was a hit, they increased how much they charged companies to place ads during the show. And if the show was a hit for long enough, they sold syndication rights to another channel. Then people could discover the show again, and maybe come back to watch it air live. In that model, the goal was to spread access to a program as much as possible to increase viewership and the number of revenue streams.
Now, in the digital age, studios have picked up a Silicon Valley trait: putting all their eggs into the basket of “increasing the number of users.” To do that, they have to create scarcity. There has to be only one destination for the thing you’re looking for, and it has to be their own. And you shouldn’t be able to control the experience at all. They should.
They’ve also moved away from creating buzzy new exclusives to get you to pay them. That requires risk and also, you know, paying creative people to make them. Instead, they’re consolidating.
Media companies keep announcing mergers and acquisitions. They’ve been doing it for a long time, but it’s really ramped up in the last few years. And these mergers are bad for all the obvious reasons. There are the speech and censorship reasons that came to a head in, of all places, late night television. There are the labor issues. There are the concentration of power issues. There are the obvious problems that the fewer studios that exist the fewer chances good art gets to escape Hollywood and make it to our eyes and ears. But when it comes specifically to digital life there are these: consumer experience and ownership.
First, the more content that comes under a single corporation’s control, the more they expect you to come to them for it. And the more they want to charge. And because there is less competition, the less they need to work to make their streaming app usable. They then enforce their hegemony by using the draconian copyright restrictions they’ve lobbied for to cripple smaller competitors, critics, and fair use.
When everything is either Disney or NBCUniversal or Warner Brothers-Discovery-Paramount-CBS and everything is totally siloed, what need will they have to spend money improving any part of their product? Making things is hard, stopping others from proving how bad you are is easy, thanks to how broken copyright law is.
Furthermore, because every company is chasing increasing subscriber numbers instead of multiple revenue streams, they have an interest in preventing you from ever again “owning” a copy of a work. This was always sort of part of the business plan, but it was on a scale of a) once every couple of years, b) at least it came, in theory, with some new features or enhanced quality and c) you actually owned the copy you paid for. Now they want you to pay them every month for access to same copy. And, hey, the price is going to keep going up the fewer options you have. Or you will see more ads. Or start seeing ads where there weren’t any before.
On the one hand, the increasing dependence on direct subscriber numbers does give users back some power. Jimmy Kimmel’s reinstatement by ABC was partly due to the fact that the company was about to announce a price hike for Disney+ and it couldn’t handle losing users due to the new price and due to popular outrage over Kimmel’s treatment.
On the other hand, well, there's everything else.
The latest kerfuffle is over the sale of Warner Brothers-Discovery, a company that was already the subject of a sale and merger resulting in the hyphen. Netflix was competiing against another recently merged media megazord of Paramount Skydance.
Warner Brothers-Discovery accepted a bid from Netflix, enraging Paramount Skydance, which has now launched a hostile takeover.
Now the optimum outcome is for neither of these takeovers to happen. There are already too few players in Hollywood. It does nothing for the health of the industry to allow either merger. A functioning antitrust regime would stop both the sale and the hostile takeover attempt, full stop. But Hollywood and the federal government are frequent collaborators, and the feds have little incentive to stop Hollywood’s behemoths from growing even further, as long as they continue to play their role pushing a specific view of American culture.
The promise of the digital era was in part convenience. You never again had to look at TV listings to find out when something would be airing. Virtually unlimited digital storage meant everything would be at your fingertips. But then the corporations went to work to make sure it never happened. And with each and every merger, that promise gets further and further away.
Note 12/10/2025: One line in this blog has been modified a few hours post-publication. The substance remains the same.
SAN FRANCISCO—With ill-advised and dangerous age verification laws proliferating across the United States and around the world, creating surveillance and censorship regimes that will be used to harm both youth and adults, the Electronic Frontier Foundation has launched a new resource hub that will sort through the mess and help people fight back.
To mark the hub's launch, EFF will host a Reddit AMA (“Ask Me Anything”) next week and a free livestreamed panel discussion on January 15 highlighting the dangers of these misguided laws.
“These restrictive mandates strike at the foundation of the free and open internet,” said EFF Activist Molly Buckley. “While they are wrapped in the legitimate concern about children's safety, they operate as tools of censorship, used to block people young and old from viewing or sharing information that the government deems ‘harmful’ or ‘offensive.’ They also create surveillance systems that critically undermine online privacy, and chill access to vital online communities and resources. Our new resource hub is a one-stop shop for information that people can use to fight back and redirect lawmakers to things that will actually help young people, like a comprehensive privacy law.”
Half of U.S. states have enacted some sort of online age verification law. At the federal level, a House Energy and Commerce subcommittee last week held a hearing on “Legislative Solutions to Protect Children and Teens Online.” While many of the 19 bills on that hearing’s agenda involve age verification, none would truly protect children and teens. Instead, they threaten to make it harder to access content that can be crucial, even lifesaving, for some kids.
It’s not just in the U.S. Effective this week, a new Australian law requires social media platforms to take reasonable steps to prevent Australians under the age of 16 from creating or keeping an account.
We all want young people to be safe online. However, age verification is not the panacea that regulators and corporations claim it to be; in fact, it could undermine the safety of many.
Age verification laws generally require online services to check, estimate, or verify all users’ ages—often through invasive tools like government ID checks, biometric scans, or other dubious “age estimation” methods—before granting them access to certain online content or services. These methods are often inaccurate and always privacy-invasive, demanding that users hand over sensitive and immutable personal information that links their offline identity to their online activity. Once that valuable data is collected, it can easily be leaked, hacked, or misused.
To truly protect everyone online, including children, EFF advocates for a comprehensive data privacy law.
EFF will host a Reddit AMA on r/privacy from Monday, Dec. 15 at 12 p.m. PT through Wednesday, Dec. 17 at 5 p.m. PT, with EFF attorneys, technologists, and activists answering questions about age verification on all three days.
EFF will host a free livestream panel discussion about age verification at 12 p.m. PDT on Thursday, Jan. 15. Panelists will include Cynthia Conti-Cook, Director of Research and Policy at the Collaborative Research Center for Resilience; a representative of Gen Z for Change; EFF Director of Engineering Alexis Hancock; and EFF Associate Director of State Affairs Rindala Alajaji. RSVP at https://www.eff.org/livestream-age.
For the age verification resource hub: https://www.eff.org/age
For the Reddit AMA: https://www.reddit.com/r/privacy/
For the Jan. 15 livestream: https://www.eff.org/livestream-age
This blog also appears in our Age Verification Resource Hub: our one-stop shop for users seeking to understand what age-gating laws actually do, what’s at stake, how to protect yourself, and why EFF opposes all forms of age verification mandates. Head to EFF.org/Age to explore our resources and join us in the fight for a free, open, private, and yes—safe—internet.
EFF is against all mandatory age verification. Not only does it turn the internet into an age-gated cul-de-sac, but it also leaves behind many people who can’t get or don’t have proper and up-to-date documentation. While populations like undocumented immigrants and people experiencing homelessness are more obviously vulnerable groups, these restrictions also impact people with more mundane reasons for not having valid documentation on hand. Perhaps they’ve undergone life changes that impact their status or other information—such as a move, name change, or gender marker change—or perhaps they simply haven’t gotten around to updating their documents. Inconvenient events like these should not be a barrier to going online. People should also reserve the right to opt-out of unreliable technology and shady practices that could endanger their personal information.
But age restriction mandates threaten all of that. Not only do age-gating laws block adults and youth alike from freely accessing services on the web, they also force users to trade their anonymity—a pillar of online expression—for a system in which they are bound to their real-life identities. And this surveillance regime stretches beyond just age restrictions on certain content; much of this infrastructure is also connected to government plans for creating a digital system of proof of identity.
So how does age gating actually work? The age and identity verification industry has devised countless different methods platforms can purchase to—in theory—figure out the ages and/or identities of their users. But in practice, there is no technology available that is entirely privacy-protective, fully accurate, and that guarantees complete coverage of the population. Full stop.
Every system of age verification or age estimation demands that users hand over sensitive and oftentimes immutable personal information that links their offline identity to their online activity, risking their safety and security in the process.
But in practice, there is no technology available that is entirely privacy-protective, fully accurate, and that guarantees complete coverage of the population. Full stop.
With that said, as we see more of these laws roll out across the U.S. and the rest of the world, it’s important to understand the differences between these technologies so you can better identify the specific risks of each method, and make smart decisions about how you share your own data.
There are many different technologies that are being developed, attempted, and deployed to establish user age. In many cases, a single platform will have implemented a mixture of methods. For example, a user may need to submit both a physical government ID and a face scan as part of a liveliness check to establish that they are the person pictured on the physical ID.
Age assurance methods generally fall into three categories:
Sometimes, you’ll be asked to declare your age, without requiring any form of verification. One way this might happen is through one-off self-attestation. This type of age attestation has been around for a while; you may have seen it when an alcohol website asks if you’re over 21, or when Steam asks you to input your age to view game content that may not be appropriate for all ages. It’s usually implemented as a pop-up on a website, and they might ask you for your age every time you enter, or remember it between site accesses. This sort of attestation provides an indication that the site may not be appropriate for all viewers, but gives users the autonomy and respect to make that decision for themselves.
An alternative proposed approach to declaring your own age, called device-bound age attestation, is to have you set your age on your operating system or on App Stores before you can make purchases or browse the web. This age or age range might then be shared with websites or apps. On an Apple device, that age can be modified after creation, as long as an adult age is chosen. It’s important to separate device-bound age attestation from methods that require age verification or estimation at the device or app store level (common to digital ID solutions and some proposed laws). It’s only attestation if you’re permitted to set your age to whatever you choose without needing to prove anything to your provider or another party—providing flexibility for age declaration outside of mandatory age verification.
The sort of parental controls found on Apple and Android devices, Windows computers, and video game consoles provide the most flexible way for parents to manage what content their minor children can access. These settings can be applied through the device operating system, third-party applications, or by establishing a child account. Decisions about what content a young person can access are made via consent-driven mechanisms. As the manager, the parent or guardian will see requests and activity from their child depending on how strict or lax the settings are set. This could include requests to install an app, make a purchase on an app store, communicate with a new contact, or browse a particular website. The parent or guardian can then choose whether or not to accept the request and allow the activity.
One survey that collected answers from 1,000 parents found that parental controls are underutilized. Adoption of parental controls varied widely, from 51% on tablets to 35% on video game consoles. To help encourage more parents to make use of these settings, companies should continue to make them clearer and easier to use and manage. Parental controls are better suited to accommodating diverse cultural contexts and individual family concerns than a one-size-fits-all government mandate. It’s also safer to use native settings–or settings provided by the operating system itself–than it is to rely on third-party parental control applications. These applications have experienced data breaches and often effectively function as spyware.
Instead of asking you directly, the system guesses your age based on data it collects about you.
Age estimation by photo or live facial age analysis is when a system uses an image of a face to guess a person’s age.
A poorly designed system might improperly store these facial images or retain them for significant periods, creating a risk of data leakage. Our faces are unique, immutable, and constantly on display. In the hands of an adversary, and cross-referenced to other readily available information about us, this information can expose intimate details about us or lead to biometric tracking.
This technology has also proven fickle and often inaccurate, causing false negatives and positives, exacerbation of racial biases, and unprotected usage of biometric data to complete the analysis. And because it’s usually conducted with AI models, there often isn’t a way for a user to challenge a decision directly without falling back on more intrusive methods like submitting a government ID.
Age inference systems are normally conducted through estimating how old someone is based on their account information or querying other databases, where the account may have done age verification already, to cross reference with the existing information they have on that account.
Age inference includes but not limited to:
In order to view how old someone is via account information associated with their email, services often use data brokers to provide this information. This incentivizes even more collection of our data for the sake of age estimation and rewards data brokers for collecting a mass of data on people. Also, regulation of these age inference services varies based on a country’s privacy laws.
ID-bound proofs, methods that use your government issued ID, are often used as a fallback for failed age estimation. Consequently, any government-issued ID backed verification disproportionately excludes certain demographics from accessing online services. A significant portion of the U.S. population does not have access to government-issued IDs, with millions of adults lacking a valid driver’s license or state-issued ID. This disproportionately affects Black Americans, Hispanic Americans, immigrants, and individuals with disabilities, who are less likely to possess the necessary identification. In addition, non-U.S. citizens, including undocumented immigrants, face barriers to acquiring government-issued IDs. The exclusionary nature of document-based verification systems is a major concern, as it could prevent entire communities from accessing essential services or engaging in online spaces.
When an image of a physical ID is required, users are forced to upload—not just momentarily display—sensitive personal information, such as government-issued ID or biometric identifiers, to third-party services in order to gain access to age-restricted content. This creates significant privacy and security concerns, as users have no direct control over who receives and stores their personal data, where it is sent, and how it may be accessed, used, or leaked outside the immediate verification process.
Requiring users to digitally hand over government-issued identification to verify their age introduces substantial privacy risks. Once sensitive information like a government-issued ID is uploaded to a website or third-party service, there is no guarantee that it will be handled securely. The verification process typically involves transmitting this data across multiple intermediaries, which means the risk of a data breach is heightened. The misuse of sensitive personal data, such as government IDs, has been demonstrated in numerous high-profile cases, including the breach of the age verification company AU10TIX, which exposed login credentials for over a year, and the hack of the messaging application Discord. Justifiable privacy and security concerns may chill users from accessing platforms they are lawfully entitled to access.
Device-bound digital ID is a credential that is locally stored on your device. This comes in the form of government or privately-run wallet applications, like those offered by Apple and Google. Digital IDs are subject to a higher level of security within the Google and Apple wallets (as they should be). This means they are not synced to your account or across services. If you lose the device, you will need to reissue a new credential to the new one. Websites and services can directly query your digital ID to reveal only certain information from your ID, like age range, instead of sharing all of your information. This is called “selective disclosure."
There are many reasons someone may not be able to acquire a digital ID, preventing them from relying on this option. This includes lack of access to a smartphone, sharing devices with another person, or inability to get a physical ID. No universal standards exist governing how ID expiration, name changes, or address updates affect the validity of digital identity credentials. How to handle status changes is left up to the credential issuer.
This is an issued token of some kind that doesn’t necessarily need network access to an external party or service every time you use it to establish your age with a verifier when they ask. A common danger in age verification services is the proliferation of multiple third-parties and custom solutions, which vary widely in their implementation and security. One proposal to avoid this is to centralize age checks with a trusted service that provides tokens that can be used to pass age checks in other places. Although this method requires a user to still submit to age verification or estimation once, after passing the initial facial age estimation or ID check, a user is issued a digital token they can present later to to show that they've previously passed an age check. The most popular proposal, AgeKeys, is similar to passkeys in that the tokens will be saved to a device or third-party password store, and can then be easily accessed after unlocking with your preferred on-device biometric verification or pin code.
With lessons pulled from the problems with the age verification rollout in the UK and various U.S. states, age verification widens risk for everyone by presenting scope creep and blocking web information access. Privacy-preserving methods to determine age exist such as presenting an age threshold instead of your exact birth date, but have not been mass deployed or stress tested yet. Which is why policy safeguards around the deployed technology matter just as much, if not more.
Much of the infrastructure around age verification is entangled with other mandates, like deployment of digital ID. Which is why so many digital offerings get coupled with age verification as a “benefit” to the holder. In reality it’s more of a plus for the governments that want to deploy mandatory age verification and the vendors that present their implementation that often contains multiple methods. Instead of working on a singular path to age-gate the entire web, there should be a diversity of privacy-preserving ways to attest age without locking everyone into a singular platform or method. Ultimately, offering multiple options rather than focusing on a single method that would further restrict those who can’t use that particular path.
In the brand new Planet Hollywood Poker Room, 48 digital rights supporters played No-Limit Texas Hold’Em in the 4th Annual EFF Benefit Poker Tournament at DEF CON, raising $18,395 for EFF.
The tournament was hosted by EFF board member Tarah Wheeler and emceed by lintile, lending his Hacker Jeopardy hosting skills to help EFF for the day.
Every table had two celebrity players with special bounties for the player that knocked them out of the tournament. This year featured Wendy Nather, Chris “WeldPond” Wysopal, Jake “MalwareJake” Williams, Bryson Bort, Kym “KymPossible” Price, Adam Shostack, and Dr. Allan Friedman.
Excellent poker player and teacher Jason Healey, Professor of International Affairs at Columbia University’s School of International and Public Affairs noted that “the EFF poker tournament is where you find all the hacker royalty in one room."
The day started with a poker clinic run by Tarah’s father, professional poker player Mike Wheeler. The hour-long clinic helped folks get brushed up on their casino literacy before playing the big game.
Mike told the story of first teaching Tarah to play poker with jellybeans when she was only four. He then taught poker noobs how to play and when to check, when to fold, and when to go all-in.
After the clinic, lintile roused the crowd to play for real, starting the tournament off by announcing “Shuffle up and deal!”
The first hour saw few players get knocked out, but after the blinds began to rise, the field began to thin, with a number of celebrity knock outs.
At every knockout, lintile took to the mic to encourage the player to donate to EFF, which allowed them to buy back into the tournament and try their luck another round.
Jay Salzberg knocked out Kym Price to win a l33t crate.
Kim Holt knocked out Mike Wheeler, collecting the bounty on his head posted by Tarah, and winning a $250 donation to EFF in his name. This is the second time Holt has sent Mike home.
Tarah knocked out Adam Shostack, winning a number of fun prizes, including a signed copy of his latest book, Threats: What Every Engineer Should Learn From Star Wars.
Bryson Bort was knocked out by privacy attorney Marcia Hofmann.
Play continued for three hours until only the final table of players remained: Allan Friedman, Luke Hanley, Jason Healey, Kim Holt, Igor Ignatov, Sid, Puneet Thapliyal, Charles Thomas and Tarah Wheeler herself.
As blinds continues to rise, players went all-in more and more. The most exciting moment was won by Sid, tripling up with TT over QT and A8s, and then only a few hands later knocking out Tarah, who finished 8th.
For the first time, the Jellybean Trophy sat on the final table awaiting the winner. This year, it was a Seattle Space Needle filled with green and blue jellybeans celebrating the lovely Pacific Northwest where Tarah and Mike are from.
The final three players were Allen Friedman, Kim Holt and Sid. Sid doubled up with KJ over Holt’s A6, and then knocked Holt out with his Q4 beating Holt’s 22.
Friedman and Sid traded blinds until Allan went all in with A6 and Sid called with JT. A jack landed on the flop and Sid won the day!
Sid becomes the first player to win the tournament more than once, taking home the jellybean trophy two years in a row.
It was an exciting afternoon of competition raising over $18,000 to support civil liberties and human rights online. We hope you join us next year as we continue to grow the tournament. Follow Tarah and EFF to make sure we have chips and a chair for you at DEF CON 34.
Be ready for this next year’s special benefit poker event: The Digital Rights Attack Lawyers Edition! Our special celebrity guests will all be our favorite digital rights attorneys including Cindy Cohn, Marcia Hofmann, Kurt Opsahl, and more!
It’s nearly the end of 2025, and half of the US and the UK now require you to upload your ID or scan your face to watch “sexual content.” A handful of states and Australia now have various requirements to verify your age before you can create a social media account.
Age-verification laws may sound straightforward to some: protect young people online by making everyone prove their age. But in reality, these mandates force users into one of two flawed systems—mandatory ID checks or biometric scans—and both are deeply discriminatory. These proposals burden everyone’s right to speak and access information online, and structurally excludes the very people who rely on the internet most. In short, although these laws are often passed with the intention to protect children from harm, the reality is that these laws harm both adults and children.
Here’s who gets hurt, and how:
Document-based verification assumes everyone has the right ID, in the right name, at the right address. About 15 million adult U.S. citizens don’t have a driver’s license, and 2.6 million lack any government-issued photo ID at all. Another 34.5 million adults don't have a driver's license or state ID with their current name and address.
Some laws allow platforms to ask for financial documents like credit cards or mortgage records instead. But they still overlook the fact that nearly 35% of U.S. adults also don't own homes, and close to 20% of households don't have credit cards. Immigrants, regardless of legal status, may also be unable to obtain credit cards or other financial documentation.
Platforms that rely on AI-based age-estimation systems often use a webcam selfie to guess users’ ages. But these algorithms don’t work equally well for everyone. Research has consistently shown that they are less accurate for people with Black, Asian, Indigenous, and Southeast Asian backgrounds; that they often misclassify those adults as being under 18; and sometimes take longer to process, creating unequal access to online spaces. This mirrors the well-documented racial bias in facial recognition technologies. The result is that technology’s inherent biases can block people from speaking online or accessing others’ speech.
Age-verification mandates most harshly affect people with disabilities. Facial recognition systems routinely fail to recognize faces with physical differences, affecting an estimated 100 million people worldwide who live with facial differences, and “liveness detection” can exclude folks with limited mobility. As these technologies become gatekeepers to online spaces, people with disabilities find themselves increasingly blocked from essential services and platforms with no specified appeals processes that account for disability.
Document-based systems also don't solve this problem—as mentioned earlier, people with disabilities are also less likely to possess current driver's licenses, so document-based age-gating technologies are equally exclusionary.
Age-estimation technologies perform worse on transgender individuals and cannot classify non-binary genders at all. For the 43% of transgender Americans who lack identity documents that correctly reflect their name or gender, age verification creates an impossible choice: provide documents with dead names and incorrect gender markers, potentially outing themselves in the process, or lose access to online platforms entirely—a risk that no one should be forced to take just to use social media or access legal content.
Age-verification systems are, at their core, surveillance systems. By requiring identity verification to access basic online services, we risk creating an internet where anonymity is a thing of the past. For people who rely on anonymity for safety, this is a serious issue. Domestic abuse survivors need to stay anonymous to hide from abusers who could track them through their online activities. Journalists, activists, and whistleblowers regularly use anonymity to protect sources and organize without facing retaliation or government surveillance. And in countries under authoritarian rule, anonymity is often the only way to access banned resources or share information without being silenced. Age-verification systems that demand government IDs or biometric data would strip away these protections, leaving the most vulnerable exposed.
Because state-imposed age-verification rules either block young people from social media or require them to get parental permission before logging on, they can deprive minors of access to important information about their health, sexuality, and gender. Many U.S. states mandate “abstinence only” sexual health education, making the internet a key resource for education and self-discovery. But age-verification laws can end up blocking young people from accessing that critical information. And this isn't just about porn, it’s about sex education, mental health resources, and even important literature. Some states and countries may start going after content they deem “harmful to minors,” which could include anything from books on sexual health to art, history, and even award-winning novels. And let’s be clear: these laws often get used to target anything that challenges certain political or cultural narratives, from diverse educational materials to media that simply includes themes of sexuality or gender diversity. What begins as a “protection” for kids could easily turn into a full-on censorship movement, blocking content that’s actually vital for minors’ development, education, and well-being.
This is also especially harmful to homeschoolers, who rely on the internet for research, online courses, and exams. For many, the internet is central to their education and social lives. The internet is also crucial for homeschoolers' mental health, as many already struggle with isolation. Age-verification laws would restrict access to resources that are essential for their education and well-being.
For many LGBTQ+ young people, especially those with unsupportive or abusive families, the internet can be a lifeline. For young people facing family rejection or violence due to their sexuality or gender identity, social media platforms often provide crucial access to support networks, mental health resources, and communities that affirm their identities. Age verification systems that require parental consent threaten to cut them from these crucial supports.
When parents must consent to or monitor their children's social media accounts, LGBTQ+ youth who lack family support lose these vital connections. LGBTQ+ youth are also disproportionately likely to be unhoused and lack access to identification or parental consent, further marginalizing them.
Age verification bills that require parental consent fail to account for young people in foster care, particularly those in group homes without legal guardians who can provide consent, or with temporary foster parents who cannot prove guardianship. These systems effectively exclude some of the most vulnerable young people from accessing online platforms and resources they may desperately need.
An age-verification system also creates acute privacy risks for adults and young people. Requiring users to upload sensitive personal information (like government-issued IDs or biometric data) to verify their age creates serious privacy and security risks. Under these laws, users would not just momentarily display their ID like one does when accessing a liquor store, for example. Instead, they’d submit their ID to third-party companies, raising major concerns over who receives, stores, and controls that data. Once uploaded, this personal information could be exposed, mishandled, or even breached, as we've seen with past data hacks. Age-verification systems are no strangers to being compromised—companies like AU10TIX and platforms like Discord have faced high-profile data breaches, exposing users’ most sensitive information for months or even years.
The more places personal data passes through, the higher the chances of it being misused or stolen. Users are left with little control over their own privacy once they hand over these immutable details, making this approach to age verification a serious risk for identity theft, blackmail, and other privacy violations. Children are already a major target for identity theft, and these mandates perversely increase the risk that they will be harmed.
The internet is today’s public square—the main place where people come together to share ideas, organize, learn, and build community. Even the Supreme Court has recognized that social media platforms are among the most powerful tools ordinary people have to be heard.
Age-verification systems inevitably block some adults from accessing lawful speech and allow some young people under 18 users to slip through anyway. Because the systems are both over-inclusive (blocking adults) and under-inclusive (failing to block people under 18), they restrict lawful speech in ways that violate the First Amendment.
Age-verification mandates create barriers along lines of race, disability, gender identity, sexual orientation, immigration status, and socioeconomic class. While these requirements threaten everyone’s privacy and free-speech rights, they fall heaviest on communities already facing systemic obstacles.
The internet is essential to how people speak, learn, and participate in public life. When access depends on flawed technology or hard-to-obtain documents, we don’t just inconvenience users, we deepen existing inequalities and silence the people who most need these platforms. As outlined, every available method—facial age estimation, document checks, financial records, or parental consent—systematically excludes or harms marginalized people. The real question isn’t whether these systems discriminate, but how extensively.
The European Commission (EC) is considering a “Digital Omnibus” package that would substantially rewrite EU privacy law, particularly the landmark General Data Protection Regulation (GDPR). It’s not a done deal, and it shouldn’t be.
The GDPR is the most comprehensive model for privacy legislation around the world. While it is far from perfect and suffers from uneven enforcement, complexities and certain administrative burdens, the omnibus package is full of bad and confusing ideas that, on balance, will significantly weaken privacy protections for users in the name of cutting red tape.
It contains at least one good idea: improving consent rules so users can automatically set consent preferences that will apply across all sites. But much as we love limiting cookie fatigue, it’s not worth the price users will pay if the rest of the proposal is adopted. The EC needs to go back to the drawing board if it wants to achieve the goal of simplifying EU regulations without gutting user privacy.
Let’s break it down.
The digital package is part of a larger Simplification Agenda to reduce compliance costs and administrative burdens for businesses, echoing the Draghi Report’s call to boost productivity and support innovation. Businesses have been complaining about GDPR red tape since its inception, and new rules are supposed to make compliance easier and turbocharge the development of AI in the EU. Simplification is framed as a precondition for firms to scale up in the EU, ironically targeting laws that were also argued to promote innovation in Europe. It might also stave off tariffs the U.S. has threatened to levy, thanks in part to heavy lobbying from Meta and tech lobbying groups.
The most striking proposal seeks to narrow the definition of personal data, the very basis of the GDPR. Today, information counts as personal data if someone can reasonably identify a person from it, whether directly or by combining it with other information.
The proposal jettisons this relatively simple test in favor of a variable one: whether data is “personal” depends on what a specific entity says it can reasonably do or is likely to do with it. This selectively restates part of a recent ruling by the EU Court of Justice but ignores the multiple other cases that have considered the issue.
This structural move toward entity specific standards will create massive legal and practical confusion, as the same data could be treated as personal for some actors but not for others. It also creates a path for companies to avoid established GDPR obligations via operational restructuring to separate identifiers from other information—a change in paperwork rather than in actual identifiability. What’s more, it will be up to the Commission, a political executive body, to define what counts as unidentifiable pseudonymized data for certain entities.
In the name of facilitating AI innovation, which often relies on large datasets in which sensitive data may residually appear, the digital package treats AI development as a “legitimate interest,” which gives AI companies a broad legal basis to process personal data, unless individuals actively object. The proposals gesture towards organisational and technical safeguards but leave companies broad discretion.
Another amendment would create a new exemption that allows even sensitive personal data to be used for AI systems under some circumstances. This is not a blanket permission: “organisational and technical measures” must be taken to avoid collecting or processing such data, and proportionate efforts must be taken to remove them from AI models or training sets where they appear. However, it is unclear what will count as an appropriate or proportionate measures.
Taken together with the new personal data test, these AI privileges mean that core data protection rights, which are meant to apply uniformly, are likely to vary in practice depending on a company’s technological and commercial goals.
And it means that AI systems may be allowed to process sensitive data even though non-AI systems that could pose equal or lower risks are not allowed to handle it.
There are additional adjustments, many of them troubling, such as changes to rules on automated-decision making (making it easier for companies to claim it’s needed for a service or contract), reduced transparency requirements (less explanation about how users’ data are used), and revised data access rights (supposed to tackle abusive requests). An extensive analysis by NGO noyb can be found here.
Moreover, the digital package reaches well beyond the GDPR, aiming to streamline Europe’s digital regulatory rulebook, including the e-Privacy Directive, cybersecurity rules, the AI Act and the Data Act. The Commission also launched “reality checks” of other core legislation, which suggests it is eyeing other mandates.
There is one proposal in the Digital Omnibus that actually could simplify something important to users: requiring online interfaces to respect automated consent signals, allowing users to automatically reject consent across all websites instead of clicking through cookie popups on each. Cookie popups are often designed with “dark patterns” that make rejecting data sharing harder than accepting it. Automated signals can address cookie banner fatigue and make it easier for people to exercise their privacy rights.
While this proposal is a step forward, the devil is in the details: First, the exact format of the automated consent signal will be determined by technical standards organizations where Big Tech companies have historically lobbied for standards that work in their favor. The amendments should therefore define minimum protections that cannot be weakened later.
Second, the provision takes the important step of requiring web browsers to make it easy for users sending this automated consent signal, so they can opt-out without installing a browser add-on.
However, mobile operating systems are excluded from this latter requirement, which is a significant oversight. People deserve the same privacy rights on websites and mobile apps.
Finally, exempting media service providers altogether creates a loophole that lets them keep using tedious or deceptive banners to get consent for data sharing. A media service’s harvesting of user information on its website to track its customers is distinct from news gathering, which should be protected.
The Commission’s use of the "Omnibus" process is meant to streamline lawmaking by bundling multiple changes. An earlier proposal kept the GDPR intact, focusing on easing the record-keeping obligation for smaller businesses—a far less contentious measure. The new digital package instead moves forward with thinner evidence than a substantive structural reform would require, violating basic Better Regulation principles, such as coherence and proportionality.
The result is the opposite of “simple.” The proposed delay of the high-risk requirements under the AI Act to late 2027—part of the omnibus package—illustrates this: Businesses will face a muddled legal landscape as they must comply with rules that may soon be paused and later revived again. This sounds like "complification” rather than simplification.
Evaluating existing legislation is part of a sensible legislative cycle and clarifying and simplifying complex process and practices is not a bad idea. Unfortunately, the digital package misses the mark by making processes even more complex, at the expense of personal data protection.
Simplification doesn't require tossing out digital rights. The EC should keep that in mind as it launches its reality check of core legislation such as the Digital Services Act and Digital Markets Act, where tidying up can too easily drift into a verschlimmbessern, the kind of well-meant fix that ends up resembling the infamous ecce homo restoration.
Axon Enterprise Inc. is working with a Canadian police department to test the addition of face recognition technology (FRT) to its body-worn cameras (BWCs). This is an alarming development in government surveillance that should put communities everywhere on alert.
As many as 50 officers from the Edmonton Police Department (EPD) will begin using these FRT-enabled BWCs today as part of a proof-of-concept experiment. EPD is the first police department in the world to use these Axon devices, according to a report from the Edmonton Journal.
This kind of technology could give officers instant identification of any person that crosses their path. During the current trial period, the Edmonton officers will not be notified in the field of an individual’s identity but will review identifications generated by the BWCs later on.
“This Proof of Concept will test the technology’s ability to work with our database to make officers aware of individuals with safety flags and cautions from previous interactions,” as well as “individuals who have outstanding warrants for serious crime,” Edmonton Police described in a press release, suggesting that individuals will be placed on a watchlist of sorts.
FRT brings a rash of problems. It relies on extensive surveillance and collecting images on individuals, law-abiding or otherwise. Misidentifications can cause horrendous consequences for individuals, including prolonged and difficult fights for innocence and unfair incarceration for crimes never committed. In a world where police are using real-time face recognition, law-abiding individuals or those participating in legal, protected activity that police may find objectionable — like protest — could be quickly identified.
With the increasing connections being made between disparate data sources about nearly every person, BWCs enabled with FRT can easily connect a person minding their own business, who happens to come within view of a police officer, with a whole slew of other personal information.
Axon had previously claimed it would pause the addition of face recognition to its tools due to concerns raised in 2019 by the company’s AI and Policing Technology Ethics Board. However, since then, the company has continued to research and consider the addition of FRT to its products.
This BWC-FRT integration signals possible other FRT integrations in the future. Axon is building an entire arsenal of cameras and surveillance devices for law enforcement, and the company grows the reach of its police surveillance apparatus, in part, by leveraging relationships with its thousands of customers, including those using its flagship product, the Taser. This so-called “ecosystem” of surveillance technologyq includes the Fusus system, a platform for connecting surveillance cameras to facilitate real-time viewing of video footage. It also involves expanding the use of surveillance tools like BWCs and the flying cameras of “drone as first responder” (DFR) programs.
Face recognition undermines individual privacy, and it is too dangerous when deployed by police. Communities everywhere must move to protect themselves and safeguard their civil liberties, insisting on transparency, clear policies, public accountability, and audit mechanisms. Ideally, communities should ban police use of the technology altogether. At a minimum, police must not add FRT to BWCs.
After a years-long battle, the European Commission’s “Chat Control” plan, which would mandate mass scanning and other encryption-breaking measures, at last codifies agreement on a position within the Council of the EU, representing EU States. The good news is that the most controversial part, the forced requirement to scan encrypted messages, is out. The bad news is there’s more to it than that.
Chat Control has gone through several iterations since it was first introduced, with the EU Parliament backing a position that protects fundamental rights, while the Council of the EU spent many months pursuing an intrusive law-enforcement-focused approach. Many proposals earlier this year required the scanning and detection of illicit content on all services, including private messaging apps such as WhatsApp and Signal. This requirement would fundamentally break end-to-end encryption.
Thanks to the tireless efforts of digital rights groups, including European Digital Rights (EDRi), we won a significant improvement: the Council agreed on its position, which removed the requirement that forces providers to scan messages on their services. It also comes with strong language to protect encryption, which is good news for users.
But here comes the rub: first, the Council’s position allows for “voluntary” detection, where tech platforms can scan personal messages that aren’t end-to-end encrypted. Unlike in the U.S., where there is no comprehensive federal privacy law, voluntary scanning is not technically legal in the EU, though it’s been possible through a derogation set to expire in 2026. It is unclear how this will play out over time, though we are concerned that this approach to voluntary scanning will lead to private mass-scanning of non-encrypted services and might limit the sorts of secure communication and storage services big providers offer. With limited transparency and oversight, it will be difficult to know how services approach this sort of detection.
With mandatory detection orders being off the table, the Council has embraced another worrying system to protect children online: risk mitigation. Providers will have to take “all reasonable mitigation measures” to reduce risks on their services. This includes age verification and age assessment measures. We have written about the perils of age verification schemes and recent developments in the EU, where regulators are increasingly focusing on AV to reduce online harms.
If secure messaging platforms like Signal or WhatsApp are required to implement age verification methods, it would fundamentally reshape what it means to use these services privately. Encrypted communication tools should be available to everyone, everywhere, of all ages, freely and without the requirement to prove their identity. As age verification has started to creep in as a mandatory risk mitigation measure under the EU’s Digital Services Act in certain situations, it could become a de facto requirement under the Chat Control proposal if the wording is left broad enough for regulators to treat it as a baseline.
Likewise, the Council’s position lists “voluntary activities” as a potential risk mitigation measure. Pull the thread on this and you’re left with a contradictory stance, because an activity is no longer voluntary if it forms part of a formal risk management obligation. While courts might interpret its mention in a risk assessment as an optional measure available to providers that do not use encrypted communication channels, this reading is far from certain, and the current language will, at a minimum, nudge non-encrypted services to perform voluntary scanning if they don’t want to invest in alternative risk mitigation options. It’s largely up to the provider to choose how to mitigate risks, but it’s up to enforcers to decide what is effective. Again, we're concerned about how this will play out in practice.
For the same reason, clear and unambiguous language is needed to prevent authorities from taking a hostile view of what is meant by “allowing encryption” if that means then expecting service providers to implement client-side scanning. We welcome the clear assurance in the text that encryption cannot be weakened or bypassed, including through any requirement to grant access to protected data, but even greater clarity would come from an explicit statement that client-side scanning cannot coexist with encryption.
As we approach the final “trilogue” negotiations of this regulation, we urge EU lawmakers to work on a final text that fully protects users’ right to private communication and avoids intrusive age-verification mandates and risk benchmark systems that lead to surveillance in practice.
Less:
When I'm writing, I write. When I'm cooking, I cook. When I'm talking to someone, I put my phone away. The constant mental juggling that felt necessary before now feels exhausting. There's something meditative about giving your full attention to a single task. [...] Less really is more. It usually takes a while to figure that out.
The allure of multitasking is, in my opinion, one of the biggest threat to actually getting things done.
My new system is, simply, no system at all. I write what I think. I delete what I don’t need. I don’t capture everything.
Refreshing take on the productivity-cult around note-taking.
Your goal is not to build a perfect system. Your goal is to solve the problem and move on. Think of productivity tools like duct tape or zip ties: functional, replaceable, and emotionally neutral.
In the same vein as the previous link. Choose simple tools. Complex ones get in the way.
We all suffer from OpenAI's pursuit of scale: A sobering podcast episode on the costs of the large-scale investment in chatbots.
I Work For an Evil Company, but Outside Work, I’m Actually a Really Good Person: This made me laugh out loud, even though the message is a sad one.
Human society is facing multiple crises. Our main method for dealing with them is technology, and we do so ignoring that similar use of technology is what brought about the crises in the first place. We use technology in attempt to remove all boundaries that stop us from getting what we want, when we want it. We are no longer able to adapt ourselves; we only know how to adapt the environment around us. Accepting our own limitations and those of nature is not considered an option, and we make every effort towards finding technological solutions that can make our desired lifestyles sustainable.
For many, the word «sustainability» is dead, either through overuse or misuse or both. I think the term still encompasses how we need to approach the way we live. Most of us are, however, not able, or willing, to see the true costs of our consumption, and therefore cannot understand what it takes to actually live sustainably. Ideally, we would not only embrace sustainability, but also frugality: Instead of using as much as our available resources allow in the long term, we should only be using as much as we actually need, which is not much.
Our attention is incessantly filled by idealisation of materialism. I do not think traditional advertising in itself is the biggest problem. The consumer lifestyle is so thoroughly ingrained in our culture that we perpetuate it ourselves through what we highlight in social media, in conversations over lunch, in parenting, in how we show appreciation of each other. The view on what we «need» is completely distorted, and we allow our personal goals and motivations to be excessively affected by what others seem to want. We are convinced that any problem can be solved by buying something, whether it is getting new clothes or having a far-away vacation.
It is a tough challenge to change our mindsets and start redefining how we approach life. Although I am outspoken about this and feel strongly about the need for change, I struggle to make significant steps towards a more sustainable lifestyle. I am, however, surprised at the general lack of desire to live differently. The effects are no longer in the future: Frequent extreme weather, shrinking biodiversity, the loss of nature1. It is long overdue that we mentally make the connection between our lifestyles and the resultant costs.
Consumers, companies, and regulators are pointing fingers at each other, and no one can agree who is responsible for taking action. It does not matter who is responsible; what matters is who can do anything about it. The answer is everyone. What I find depressing is that many have no faith in the actions of individuals. Any large reform starts with human beings willing to speak up. Personally, however, what motivates me the most is not whether my actions can actually lead to change, but rather that I want to contribute as little as possible to the destructive exploitation of the planet and of the less fortunate people who are paying the price for the over-consumption by the richest.
To contribute to greater visibility of people and projects that are aiming to change the current trajectory, I am collecting online resources that inspire and inform on the topic of sustainable living: Sustainability. In this context, I use the terms "sustainability" and "sustainable" to mean having respect for our own and nature's limitations, ensuring that humans and nature can co-exist indefinitely with minimal harms and losses. Due to my line of work, I have a bias towards focusing on digital technology, but I also collect resources from other domains.
Achieving meaningful change is virtually impossible within the current system, and grassroots movements outside it is crucial to find new ways forward. Hopefully, many small projects can snowball into a change of the tide.
The Guardian: Europe losing 600 football pitches of nature and crop land a day. It is incredible that we say we are «losing» nature, when we are actively eradicating it in favour of «development». ↩
Recently I read an article about how the average sentence length in Norwegian fiction books has gone down by 20% since the 1980’s. The results were based on over 7000 fiction books from 1980 to 2020. The study itself did not conclude on the cause of this, but I immediately thought of our decreasing attention spans, excessive smartphone use, and the decline of long-form reading. An author who was interviewed about the results alluded to similar causes, but she also speculated whether the popularity of audiobooks was a contributing factor, because authors adapt their writing to sound good when spoken aloud.
The trend of shorter sentences made me think of how the formatting of text affects both our writing and reading, and I remembered a blog post I read recently. It was written many years ago, by someone who appeared to care little about conventional text formatting. The blog post had no linebreaks whatsoever; it was a single block of text. One massive paragraph, with no entry points for eyeballs wanting to skim through the text, searching for keywords to latch onto. From the style of writing, it did not appear intentional; it seemed more like an oversight. As such, I was forced to read through the whole text from start to finish to even figure out what the text was about. While doing so, I gradually got into a state of reading I seldom enter while surfing the internet. Reading a text with no natural breaks anywhere requires more focus than usual, but it was particularly unfamiliar when sitting in front of a computer screen. The blog post stood in stark contrast to the way a lot of online text looks these days.
I realized then how much the formatting of text affects the reading process. It may sound obvious, but it’s worth considering how different online writing is from printed books, and how it has been changing during the internet era. The emergence of generative language models has brought another wave of change. Many are now familiar with the «ChatGPT-format»: Extensive use of lists, where the first phrase of each bullet point is put in bold text. This type of text formatting lends itself easily to skimming instead of getting properly engaged with the text. It takes little effort to produce, and little effort to consume. I’m not sure why this particular format has become so prevalent for chatbots. It could be an artefact of the data they were trained on, but I think it is more likely to be a conscious decision by the developers to reinforce the easily digested format of lists. Easily digested does not mean healthy, though.
More concise writing can easily be observed in a lot of online newspapers as well: Not only are the sentences short, but the paragraphs are brief, often just one or two sentences. Perhaps it is done this way to be easier to read on mobile screens. I am sure most online magazines track their users thoroughly enough to know exactly how long they spend reading each article and how far they get into the text, giving them ample opportunity to adapt the format to get the most clicks and time spent looking at their content and ads.
I can observe how this change is affecting me. The tendency to favor bullet point lists over continuous sentences, the habit of writing shorter paragraphs, and the desire to highlight my most important points with bold text. These dispositions appear in everything from emails to research papers. Anything to increase the likelihood of getting the message across to the reader, before their attention is sucked away to something else. Context matters, of course, and in some cases bullet points are exactly what you need. But I realize now that this type of writing often may be doing both the writer and the reader a disservice. When I write, the bullet points are useful for superficial thinking. The lists are for itemizing, categorizing, and summarizing. The long paragraphs are what I need to go deeper. The act of stringing together longer sentences is what forces me to formulate an idea coherently, and not just present a set of concepts or ideas. On the other end, the reader sometimes needs to be forced into the mode of focused reading in order to properly understand and digest the message. A bullet point list, a set of short paragraphs, a few words in bold; all of these are opportunities to jump ahead and skip important parts that end up in the background.
It is worth upholding the notion that important ideas need the long-form format to be properly understood. For writers, it is better to have your text understood properly by one reader than skimmed through by many. Dare to write longer when everyone else uses easily digested lists. As a reader, choose to get absorbed in a longer piece instead of glancing through multiple. Quality over quantity goes a long way in both writing and reading.
Earlier this year I had the chance to realise a project I’ve thought about for several years: Skiing the couloir called Storskæra on Allmannberget, a mountain in Oppdal, Norway. The couloir is very visible from the highway going through Oppdal, and I’ve thought about it virtually every time we’ve been passing through.
I remember when I first heard about people skiing down Allmannberget, about 15 years ago (though the first descent was probably much, much earlier than that). I was not familiar with steep skiing, and I couldn’t comprehend that it was possible to get down on skis in that sort of terrain. At the time, I wasn’t sure if I believed that people were doing it.
Now, on the other hand, I realise that Storskæra is not very extreme in the context of steep skiing, although you still need to know what you’re doing. While the couloir is over 50 degrees in the uppermost part, most of it is between 35-45 degrees.
In general, I can’t be too picky about the snow conditions because of certain constraints on my calendar. It’s however vital that there’s a low avalanche risk. In late April, the snow was generally stable and safe in the area, but with regards to skiing conditions, the temperature was not on my side. The cold weather had given the snow a hard crust. When I got to Allmannberget, the snow was hard already from the bottom of the couloir, and it got worse the higher I got. I always prefer to walk up the couloir before skiing if possible, so I know what I can expect when going down. This time, I had to wear crampons the whole way up, meaning it wasn’t going to be a very enjoyable run down again.
On my way up.
The couloir is north-northwest facing, so it’s usually preferable to go there late in the day, to let the sun soften up the snow. If I’d had the time, I would have waited a while before skiing. The top part would probably have been much the same anyway because of the cold wind up there, but the lower segments would most likely have had more slushy than crusty snow.
View towards Oppdal.
Almost at the top.
After getting up to the summit ridge, I made a quick visit to the mountain’s highest point before heading back to the top of the couloir. Looking down the chute, knowing that the snow would be hard and icy most of the way, I took a minute to consider other options for going down. I decided however that I didn’t want to go home with unfinished business. I felt confident I could get down safely, it just wouldn’t be as fun as it could be with better snow. The main goal of the day was to do something challenging, and this would qualify.
Ready to descend.
I had to practically downclimb the couloir’s steep entrance while wearing skis, with an ice axe in each hand. The snow was almost as hard as ice, and I struggled getting purchase with my skis. Since I often ski with dogs, I seldom sharpen the edges, reducing the risk of cutting any legs or paws. I deeply regretted skipping that at this point. Further down I could stop hacking at the snow with axes, and use only my legs to control the descent. Carefully making turns, but also just sliding sideways for many meters at a time, to stay in control. Not a pretty sight, but efficient. There were also two sections of exposed rock in the couloir, where I had to remove my skis and move down some scrambling terrain.
Own my way down.
After descending the full length of the couloir on icy conditions, I was tired both physically and mentally, but happy to have completed a long-time goal. As a bonus, I did the neighbouring couloir called "Haninj" directly afterwards. It’s not as long or spectacular, but the skiing was more fun since the snow started getting softer.
Ascending "Haninj".
The main take-away from the day was observing the progress, going from not believing something was doable, to actually doing it. Because of that, Storskæra was a small personal milestone. Although it’s not much of a challenge for experienced skiers (it definitely pales when compared to skiing down a climbing route), it’s worth appreciating the effort it takes to improve.
I often pass through Oppdal, meaning I can enjoy the sight of Storskæra frequently. I hope to repeat it someday when conditions are better, but I also have a list of other couloirs that has to be explored first.
Digital social media is hard. There is so much information, creative output, advertisement, and attention-seeking. The «algorithm» is there to help us figure it all out, by sorting, filtering, and prioritising the content in the feed of your chosen social media.
Algorithmic curation gets a lot of critique, and most of it is well-deserved. It’s a tool which on the surface tries to help you find what you’re really looking for in all the noise, but in reality it’s being used to maximise time spent browsing on any given platform, hoping you will not only watch ads, but also click on them. However, this type of curation is not without merit, because the flood of digital information is so large on most platforms that they would be unusable without any sort of filtering.
The "algorithm" is not the only way of solving this problem though. With decentralised social media protocols such as the Fediverse, where Mastodon is a popular platform, you can to a much larger degree control what you get. No additional content will be injected in your feed. No ads, no «suggested for you», no «you may also like». That makes it easy to stifle the noise by simply following fewer people.
For me, the solution to the issue is to simply avoid social media as much as possible, and rely on direct personal recommendations and tips. Those around me have gotten used to sending me links to news articles, podcasts, events, blog posts etc, because they know I don’t discover things through Facebook and Instagram.
I also get a lot of good recommendations through other personal websites. I would like to highlight Tim Hårek and Baldur Bjarnason for regularly sharing links that I usually find worth a read. Tim shares interesting reads in his monthly write-ups, and Baldur has separate posts with collection of links. I also publish «link posts» from time to time with some of the interesting things I’ve found around the web, like this one.
This post ends with an encouragement to make sharing stuff more human and organic. We need to help each other find out what’s worth spending time reading, and not let an automated algorithm decide it for us.
Black Lake Race is a quite new addition to the race calendar in Norway. I had planned to participate in its first edition last year, but I couldn’t go because of more pressing matters at home. The route, with its steep terrain and long off-trail sections, is a good match for me, and there are not many races in Norway with this kind of terrain. That’s why the race was on my priority list for 2025.
The race doesn't have much PR, but it's grounded in the local community with a commitment to true mountain running. The extreme terrain, compared to most trail and mountain races in Norway, doesn’t attract the big masses. Last year Kilian Jornet, who lives in the area, participated and won. I was hoping he would turn up again this year, giving the rest of us a chance to measure up. I raced against him in Isfjorden Skyrace last year, taking second place behind him, and I enjoyed the extra edge the race gets when a true elite runner is at the start line. Unfortunately, he didn’t join this time.
The weather forecast for the day was hot and sunny, which is not in my favour. Earlier this year I’ve done some heat workouts, which seems like the big trend of the year among endurance athletes. I could definitely benefit from heat adaptation, but my daily schedule and other responsibilities makes it hard to prioritise the extra hassle of putting on and off many layers of clothes. In my mind, I get more benefit from spending those minutes (which accumulates to hours) actually training.
My goal was to spend about 3:40 on the course, adding 20 minutes to the course record. I planned to consume three bottles of Maurten 320 during the race, giving me about 240 grams of carbohydrate, and drinking at least 3 liters of water. I know my strengths well enough to portion out my energy in a strategic manner, holding back a bit on the uphills and keeping a good momentum in the downhills.
I started out in a controlled tempo, and got a gap of a couple of minutes by the first checkpoint at 5 km. The next uphill was the main challenge of the race: An ascent of 850 meters of elevation gain in only 2 kilometers. The first part consists of a combination of marshes and bushes. The locals claim there is a trail there, but I couldn’t see it. This particular section is the most brutal part of the course; not only is the slope punishing, but you have to fight for every step because of the soft, squishy ground or bushes up to your thigh.
Further up, the real mountain terrain takes over. I got into a good rhythm, and reached the first summit with a comfortable lead. I could enjoy the scenery as the race course passes over three ridges in rocky terrain, all while having an excellent panorama to the surrounding mountains. The two main peaks of the route, Trollstolen (1141 masl) and Trolltind (1241 masl) are spectacular in themselves, but the view of the serrated ridge between Trolltind and Svartevasstind is the eye-catcher of the landscape. It was nice to be able to appreciate the views, but I got a bit too comfortable running alone, and didn't manage to stay fully in race mode and keep pushing.
The second checkpoint is located between ridge number two and three, and here we joined the 20 km course, which was a shorter option of the Black Lake Race. I got a small motivational boost to speed up by chasing the leader of the shorter race, Emelie Forsberg, up to Trolltind. Down from the last summit, Blåtinden, the markings got very sparse, and for a long time I couldn’t spot a single marker. I tried my best to just follow the GPS track on my watch, but I got into a particularly gnarly terrain on an unnecessary detour. I got frustrated and lost a bit of time. Finally I came onto a trail again, and for the last 6 kilometers I tried to speed up for a strong finish.
I ended up spending 3:51 on the course. Although I’m happy about finishing in first place, the performance was lacking. My original estimate of 3:40 should have been within reach with more motivation to push during the race. In terms of nutrition and hydration, I followed my plan and drank three bottles of Maurten 320 and in total 3.5 liters of water, which felt sufficient to keep a stable pace.
I told the organizers that Black Lake Race is my favourite race course in Norway, and I meant it. I don’t think you’ll be able to find a current Norwegian race that is more pure skyrunning than this. I’m planning to go back next year and improve my time!
Podium of Black Lake Race 2025.
A few interesting articles I've read around the web:
The life of a pessimist is easy but dreary. The life of an optimist is hard but exciting. Pessimism is easy because it costs nothing. Optimism is hard because it must be constantly reaffirmed. In the face of a hostile, cynical world, it takes effort to show that positivity has merit.
An observation that persistently crops up in research into the use of Large Language Models is that regular use seems to have a diminishing effect on your critical thinking ability, even in studies that are outright set out to look for justification for using LLMs. […] The very point of “AI” tools is to think less about what you do, think less about the arguments you make, and generally put less thought into your work and your life as a whole. That’s what it’s for.
Marcus Aurelius, writing in his private journal that we now call Meditations, returned constantly to the idea of limits. He didn’t prescribe grand goals. He reminded himself what not to do: Don’t lie. Don’t whine. Don’t be ruled by impulse. The Stoic path is constraint-oriented. It avoids the seduction of outcomes.
LLMs give us ventriloquism in reverse. The mechanical dummy speaks through your mouth.
Suppose Schopenhauer is right that life boils down to a flight from either boredom or pain. Insofar as the vast material abundance of wealthy, industrialised society has had an analgesic effect (there is simply less physical pain than in the past, before fluoride and anaesthesia and sedentary lives), it would seem to have solved one problem only to amplify the other. In place of pain, we have ennui, the quintessential modern condition. It follows directly from overabundance: an endless stream of video “content” or chocolate cake or edibles or any other indulgence cannot deliver lasting satisfaction. Everything gets old eventually, leaving one to grope around for the next fix.
Armed with generative AI, a B student can produce A work while turning into a C student.
Additionally, I have a bonus link for Norwegian speakers (though I recommend it to everyone else as well, if you have a convenient translation tool):
I prosessen med å bli verdens mest digitaliserte land risikerer vi å få verdens mest overvåkede innbyggere. (In the process of becoming the world’s most digitalised country, we risk getting the worlds most monitored people.)
The progress of AI is interesting and important to follow, but current discussions around it are too focused on the increasing benchmark performances and if machines can ever get smarter than humans (superintelligence). Counter-arguments to the promise of achieving so-called «superintelligent» computer programs can typically take the form of «humans are still better than AI at this», or «AI can never replace humans at that», but they are missing the point. Asking whether AI can outperform humans is the wrong question. The real question is: What are appropriate uses of AI? Or simply, what do we actually want to automate?
One of the main reasons for developing AI technology is to increase efficiency through automation. We want the computer to solve certain tasks on its own. The problem we now face is that automation technology is so advanced that there seems to be no limit to what we can automate. Given enough time, money, and resources, we can make machines that are, or at least appear to be, more efficient than humans for virtually any task, especially if we only evaluate them on task efficiency and ignore any ripple effects. This of course depends entirely on what metrics you use to evaluate efficiency, but any concretely defined metrics can theoretically be optimised for when training an AI model.
From a societal point of view, it is pointless arguing whether AI or humans are better at a certain task without first asking whether the task is suitable for AI in the first place. This topic was brought up already in 1976 by Joseph Weizenbaum in his book Computer Power and Human Reason, and has only become increasingly relevant. Generative AI-models’ ability to process and generate text leaves practically no area of society out of reach. This is exactly when we need to not only worry about performance and reliability of AI systems, but how we actually want to use them.
Consider for example AI chatbots for keeping lonely elderly people with company. This suggestion comes not only from outside the target group but also from elders themselves. With limited personnel and resources for elderly care, why not have an AI companion that can easily be individually personalised and is always available? There is no doubt that the current AI chatbots can generate human-like, empathic conversation, but do we really want to use computers as company for our elders?
The question is hard, because it forces us to search for our common values. Does it matter if we replace care-givers with machines? If studies show that it works (according to certain metrics) and it’s difficult to employ enough people, then why not?
While a chatbot might be better than nothing, we can certainly do much better. Interpersonal relationships are a cornerstone of both society and our individual lives, and of all the things we would like to automate, it should be far down the list. The challenge of human loneliness is one area where political solutions must take precedence over technological ones, such as investing in care workers and community programs. And similar dilemmas appear when AI-systems are used as therapists, companions, or other forms of direct replacements for human relations.
While Weizenbaum had the luxury of discussing hypothetical scenarios, we are facing concrete challenges. When new technologies appear, we can always choose whether to use them or not. The answer depends not only on the capabilities of the technology, but what we value as a society. Once the choice is made, it’s often hard to reverse it.
Just because we can automate human connection doesn’t mean we should. Let’s use machine intelligence to automate machines, not humans.
I recently came across a review of Sky, an upcoming AI automation app for macOS. The technical aspects piqued my interest and made me click the link, because I work with AI engineering research. It was the societal implications, however, that made me read the whole thing from start to finish.
In short, Sky is an AI-assistant that integrates deeply into macOS, and can interact with any app, independent of whether the app has been developed for AI-integration or not. The features of Sky are impressive, and based on the review it looks like the developers have managed to create something quite innovative.
However, the promises of «natural computing» left me less daunted than the potential issues such software creates.
I find it incredible that privacy isn’t mentioned a single time in this review. It’s the first thing that pops up in my head when reading about the tool, and it should be the first thing to check before even thinking about using something with this level of access.
The tool’s core strength, being able to ingest anything you do on your computer, also becomes its greatest weakness: Any data you feed it will be sent to the LLM-provider powering the tool (either OpenAI or Anthropic, according to the review).
One thing is the user’s own data, but what about all those «skyshots» (Sky’s advanced screenshots that captures what you’re doing on the computer) of other people’s messages, among other things, as shown in the main demo on Sky’s homepage? While screenshotting and saving people’s messages always has been possible, this app makes sharing such data with a third party a part of its core functionality. A Sky user will be accepting the privacy policy of both Sky and OpenAI/Anthropic on behalf of anyone whose data appears on the screen, likely including info such as names and email addresses.
Improved performance of local LLMs may enable a more private version of this tool, but let’s face it: The best models will always be the more power-hungry variants in the cloud, which also are the ones that generate the most income for providers.
Another thought that came to mind when reading the review was how sometimes instructions for an LLM end up being longer than what you want it to produce. In one example, the reviewer wants to send a link and a summary of it to a friend, and after receiving the first LLM-written draft, he gives the following command:
also, start the message by saying «this is cool» followed by the link
Why bother giving a command that is twice the length of what you want to write? This particular example could of course be explained by several factors (f. ex., «I was already writing in the command prompt, and it would have taken more time to move my cursor for editing the text»).
However, I don’t think this type of instruction-giving uncommon, and I’ve seen similar cases of people asking using AI-assistants for programming. Why would we give an instruction when doing the action ourselves takes equal (or even less) amount of time? Perhaps it is something about the cognitive difference between giving an instruction and having to perform an action. It seems to me that there is something comfortable about sitting in the driver’s seat and command, rather than do the actual work.
While this is seemingly not a big issue and just a nitpick in the context of the review, it becomes more serious when considering the large energy consumption of LLMs. While prompting a chatbot has a minuscule carbon footprint compared to many other things many of us do in our daily lives (driving a diesel car, taking long hot showers, etc), the relative benefit of running AI-automation is probably not worth the cost for such extremely trivial tasks.
The allure of automation and increased efficiency is tempting, and part of me gets intrigued and excited when hearing about such technical advances. This is in spite of the fact that I have a quite analog lifestyle, and have few processes I would need to automate with this tool. The concept of computers is after all making data-processing more efficient, so why wouldn’t we take every opportunity to automate? A lot of computer work is simply boring and unrewarding, and not every user interface is as good as it perhaps should have been. Shouldn’t we use AI to save us some time?
It’s not only about privacy, or the increasing need for computational resources which puts pressure on energy, water and hardware supply. These are important issues, but I also worry about the aspect of letting automation sneak into the realm of inter-personal relationships. I think it’s essential to keep communication between us authentic and genuine.
I still get surprised by how many demos for AI-automation showcase how you can automate communicating with other people, including friends and family. Why do we need to make that more efficient? It appears to be fostering the same sort of friendships that having hundreds or thousands of «friends» on Facebook does.
In the introduction, the reviewer writes:
As soon as I saw a demo, […] I knew Sky was going to fundamentally change how I think about my macOS workflow and the role of automation in my everyday tasks.
Well, as soon as I read this review, I knew that this kind of automation tools will kill privacy, needlessly accelerate our daily lives even further, and take the «personal» out of personal communication. That is, if it catches on. And I suspect it will, in some form or another.
When considering to buy a digital device, service or tool (or any type of technology for that matter), it is worth thinking about the following:
When deciding on a specific product, particularly for physial devices, there are another set of questions to consider:
Taking these aspects into account can help you choose more sustainable, privacy-friendly and humane technology.
Not only must we be mindful of what effects certain technologies have on our minds and our society, especially when deployed on an enormous scale, but the rapid obsolescence of digital devices is simply extreme.
The considerations above have led me to replace dozens of apps with a paper notebook, and I often carry a feature phone and a camera instead of a smartphone. The downstream effects include a drastic reduction in the time I stare into a screen, being more present in the physical world, and having my focus protected against the attention mining of internet-abusing companies.
Choose wisely.
On the other you have the people who are realising that maybe the solution is not to recreate social media but rather to abandon it and go back to a more deliberate way to be social online, using personal sites, small forums, emails, and other “traditional” tools. I’m obviously part of this second group.
There is a small but significant number of people who really like generative AI. This isn’t the same as the number of people who’ve used (or claim to have used) genAI, but a subset of superfans. I tend to meet two kinds of people in this category: enthusiastic senior leaders who have been tasked with doing something innovative but might not be hands-on users, and more junior people who are enterprising and computer-literate and want to try something new. Notably, some of the people in the second group might have a natural affinity for prompt writing but not enjoy another part of their job – like writing long documents or emails – and so are pleased to outsource those tasks to AI.
But let’s be clear, it’s not “humanity” as a whole that’s exploiting the earth. It’s the top 0.1% exploiting everything and everyone else. The majority of humanity is just as exploited as the forests, oceans, and animals. Choosing exploitation means embracing a brutal, materialistic race to the top. A system where everyone except a tiny minority loses.
Our mind, not the technology, is the bottleneck. We need to care about our minds. To dedicate time to think slowly and deeply.
Forrige uke var jeg så heldig å få fortelle om vår forskning innen mer bærekraftig kunstig intelligens på en av Norges største populærvitenskapelige radioprogram, nemlig Abels Tårn på NRK!
Først og fremst var det veldig gøy å få være med, og det var en flott sjanse til å fortelle om hva jeg holder på med til daglig. Ikke mange er klar over at det jobbes aktivt i Norge med forskning for å adressere klimaavtrykket til KI, og på min forskningsgruppe (Green IoT) leder vi forskningen innen «grønn KI» i EU-prosjektet ENFIELD1. En av de tingene vi driver med akkurat nå er å kartlegge energiforbruket til ulike generative språkmodeller, og hvordan ulike faktorer påvirker både energieffektiviteten og ytelsen.
Er det en ting som er sikkert, så er det at det er mange estimater, usikre tall, og manglende kildehenvisninger ute og går når det gjelder strømforbruk og klimaavtrykk KI-modeller, spesielt språkmodellene bak ChatGPT. Generelt sett har vi så lite kunnskapsgrunnlag om modellene til OpenAI, at vi kan ikke si med sikkerhet hvor mye en ChatGPT-spørring (-prompt) koster i verken strøm, vann eller karbonutslipp. Likevel må man innimellom prøve å konkretisere forbruket litt ved å ta i bruk slike estimater, selv om det ikke alltid er så lett å forklare konsist hvordan disse beregningene er gjort. Utviklingen går så fort vil de fleste estimater foreldes ganske raskt, med nye språkmodeller som dukker opp.
Det overordnede bildet blir likevel stående: Generativ KI har et signifikant ressursforbruk som vi er nødt til å ta hensyn til når vi vurderer hva vi skal bruke denne teknologien til. Regulering og åpenhet trengs for å adressere problemet.
Jeg håper intervjuet kan være med på å gjøre KI litt mindre mystisk, litt mer håndfast, og å forklare hvorfor klimaavtrykket til generativ KI er så stort sammenlignet med andre former for digital teknologi.
Tusen takk til radiovertene i Abels Tårn, Torkild Jemterud, Kristina Aas og Hanne Aass, for å invitasjonen og en meget hyggelig samtale rundt temaet!
Hør på episoden her: Abels Tårn 7. feb 2025 (start fra 29 min)
As AI is discussed in all kinds of media, futuristic images abound in news papers, presentation slides, blog posts, and social media. The problem with many of these images is that they muddle the impression of what we are actually talking about when we discuss AI, which in many cases are simply a huge number of additions and multiplications being performed on a computer (yes, that's what the chatbot you're using as an orcale really is).
I just discovered a handy resource when you need illustrative images for AI-related topic: Better images if AI. As the front page of the website says (emphasis mine):
Abstract, futuristic or science-fiction-inspired images of AI hinder the understanding of the technology’s already significant societal and environmental impacts. Images relating machine intelligence to human intelligence set unrealistic expectations and misstate the capabilities of AI.
The website is run as a collaboration between multiple organizations (BBC, We and AI, Leverhulme CFI), where they share more meaningful AI-related images. They are free to use as long as an attribution is included.
Now you don't even have to waste electricity on asking Midjourney to generate illustrations for your slides!
Another collection of interesting reads from around the web:
I recalled Zadie Smith’s essay “Fail Better,” in which she tries to arrive at a definition of great literature. She writes that an author’s literary style is about conveying “the only possible expression of a particular human consciousness.” Literary success, then, “depends not only on the refinement of words on a page, but in the refinement of a consciousness.”
I'm happy to see that there is an increasing focus on the challenge regarding AI and sustainability. Not only because I'm a researcher trying to make AI "greener", but because AI's booming resource consumption is starting to become a real problem. The last few months I have seen many articles bring up some of the issues, and I would like to highlight a few of them.
The Atlantic writes about "Microsoft's hypocrisy on AI"1, describing how Microsoft claims to take responsibility for climate action, while still having large scale cooperation with fossil-fuel companies. While the article's focus is mainly on the application of AI, I think the following observation about AI's own climate footprint is important:
The idea that AI’s climate benefits will outpace its environmental costs is largely speculative, however, especially given that generative-AI tools are themselves tremendously resource-hungry. Within the next six years, the data centers required to develop and run the kinds of next-generation AI models that Microsoft is investing in may use more power than all of India. They will be cooled by millions upon millions of gallons of water. All the while, scientists agree, the world will get warmer, its climate more extreme.
There are many possible ways in which we can use AI for limiting carbon emissions, but widely deployed chatbots and image generators are not among them. We have to stop generalizing the benefits of one AI application to support any type of AI usage.
The article is based partly on an interview with Holly Alpine, previously Microsoft's Sr. Program Manager of Datacenter Community Environmental Sustainability. She has written her own account on why she left the company2, which gives additional insight to the problem:
Internally, discussions highlighted the most significant application of generative AI for the oil and gas industry: optimizing exploration activities. AI was celebrated as a “game changer” and the key for the fossil fuel industry to continue to be competitive.
Again, AI can be used to fight climate change, but it certainly can be used to hasten it as well.
There is no doubt that the enormous investment in AI, and particularly generative AI, is driving energy consumption and emissions up for many companies. NPR reports that both Google and Microsoft have significantly increased carbon emissions3. Google's emissions are up by 48% since 2019, and Microsoft's have increased by 29% since 2020.
This is despite the fact that both companies have pledged to be carbon-neutral by 2030. In both cases, the need for scaling up infrastructure to support AI is given as a reason for why emissions are soaring.
I was lucky enough to be interviewed about the energy consumption of large language models in the online news magazine Heatmap News4. Slightly less known than The Atlantic and NPR, but even so, I cherish every opportunity to talk about my research in more public channels.
I tried to drive home the point that generative AI is in general very expensive, and that we need to think hard about when and where it is actually worth using it. It was mainly the first point that made it through to the final article, but the second one is just as important.
I commented specifically on the new o1-models from OpenAI, which use so-called "chain-of-thought" to increase their performance in problem solving. While the effort to reduce the hallucinations of LLMs is laudable, the new models are almost certainly much more expensive than previous models. To quote myself:
As Erik Johannes Husom, a researcher at SINTEF Digital, a Norwegian research organization, told me, “It looks like we’re going to get another acceleration of generative AI’s carbon footprint.”
Let's hope we have some breakthroughs soon to make models more efficient, not more resource-demanding.
There is an underlying challenge with working towards making AI (or any type of technology) more energy-efficiency: The cheaper it gets to use something, the more likely we are to use more of it.
I'm worried that our efforts to make AI "greener" will only lead to more unecessary consumption.
Nevertheless, increased awareness is a good beginning for making a change.
Hao, K. (2024, September 13). Microsoft’s hypocrisy on AI. The Atlantic. https://www.theatlantic.com/technology/archive/2024/09/microsoft-ai-oil-contracts/679804/. Archive.org: https://web.archive.org/web/20240913122346/https://www.theatlantic.com/technology/archive/2024/09/microsoft-ai-oil-contracts/679804/. ↩
Alpine, H. (2024, September 18). I loved my job at Microsoft, but I had to resign on principle. Here’s why. Fortune. https://fortune.com/2024/09/17/microsoft-environment-oil-fossil-fuel-expansion-climate-change/. ↩
Kerr, D. (2024, July 12). AI brings soaring emissions for Google and Microsoft, a major contributor to climate change. NPR. https://www.npr.org/2024/07/12/g-s1-9545/ai-brings-soaring-emissions-for-google-and-microsoft-a-major-contributor-to-climate-change. ↩
Zeitlin, M. (2024, September 17). What Does OpenAI’s New Breakthrough Mean for Energy Consumption? Heatmap News. https://heatmap.news/technology/openai-o1-energy. Archive.org: https://web.archive.org/web/20240917231307/https://heatmap.news/technology/openai-o1-energy. ↩
A couple of months ago I decided it was time for a digital deep clean of my laptop. It was close to three years since last time, and over that period a lot of unnecessary software and data has aggregated in the nooks and corners of the system. The easiest way to achieve a clean system is to reinstall the operating system, which also gave me a chance to upgrade Ubuntu from 22.04 to 24.04.
When setting up a new system, one of the first things you want to do is to install and configure your most used programs.
As many other programmers and Linux users, I have a dotfiles folder that contain my precious configuration files for all the vital utilities I use.
For me, the most important ones are bash, vim and tmux.
It's now eight years since I entered a computer terminal1 for the first time, and over that period a lot of things have accumulated in my dotfiles. I have been using different Linux distros, macOS, and Windows Subsystem for Linux, and have made my configuration adaptable to several different setups.
At the same time, I like to think of myself as a minimalist, in some sense. I like to keep my system as fast and uncluttered as possible. I'm not sure if my dotfiles reflect this, but I know I'm quite restrictive compared to some "vim maximalists".
My .vimrc has grown to ~500 lines, which is excluding the vimscripts, plugins and templates I use.
I have a .bashrc of ~400 lines, and a .tmux.conf of ~60 lines.
In addition to that, I have 65 bash scripts, counting ~5500 lines in total. These are not really configuration files, but I include them in my dotfiles as a part of my bash setup.
My dotfiles are by no means bloated compared to other software I use regularly, but I wondered: How much of this I actually need?
With a newly installed operating system, I thought it would be an excellent opportunity to start from scratch, and only add back things I actually felt the need for.
Now, after about 7 weeks since the clean install, I think it's time to assess the status. What parts of my dotfiles did actually make it on to the next phase?
In bash, I added back some lines quite quickly, mainly history settings and aliases. I use many shortcuts all the time, and they are firmly etched into my muscle memory:
# Use vi keybindings on command line
set -o vi
# Prompt appearance
export PS1="\[\e[31m\]\u\[\e[m\]@\[\e[32m\]\h\[\e[m\] \[\e[34m\]\w\[\e[m\] \\$ "
# Adjust history size
HISTSIZE=20000
HISTFILESIZE=20000
SAVEHIST=20000
# Ignore duplicate entries in the history
HISTCONTROL=ignoredups:erasedups
# The following lines are needed to save history from multiple sessions
# append history entries..
shopt -s histappend
# After each command, save and reload history
PROMPT_COMMAND="history -a; history -c; history -r; $PROMPT_COMMAND"
# Aliases
alias x='exit'
alias ..='cd ..'
alias ...='cd ../..'
alias ....='cd ../../..'
alias .....='cd ../../../..'
alias ......='cd ../../../../..'
alias v='vim .'
alias ll='ls -AhlF --color=auto' # human-readable file size
alias la='ls -A'
alias l='ls -CF --color=auto'
alias duh='du -h -d 1' # Show file human-readable file sizes
alias df='df -h' # Show disk usage with human-readable sizes
alias cp="cp -i" # confirm before overwriting something
alias grepr="grep -rn . -e"
alias g='git status'
alias ga='git add'
alias gc='git commit'
alias gcm='git commit -m'
alias gd='git diff'
alias gp='git push'
alias gpl='git pull'
alias gl='git log --graph --oneline' # show git log as graph
alias gcred='git config credential.helper store'
alias gitmail='git config user.email'
alias gitname='git config user.name'
alias gclone='git clone'
alias gall="find . -maxdepth 1 -mindepth 1 -type d -exec sh -c '(echo {} && \
cd {} && git status -s && echo)' \;" # check git status of all subfolders
# git pull on all subfolders:
alias gplall="find . -maxdepth 1 -mindepth 1 -type d -exec sh -c '(echo {} && \
cd {} && git pull && echo)' \;"
The configuration file itself was reduced down to about 25% of its previous size, but the biggest difference is for the bash scripts: Out of 65, none have made their over to my freshly installed system.
All the time I spent writing scripts to automate certain actions seems to have gone to waste (although I probably learned a lot in the process). I guess I miscalculated how much I would actually use those scripts in my daily work.
For tmux, I needed about half of my original configuration:
# Remap prefix to Control + a
set -g prefix C-a
# Bind 'C-a C-a' to type 'C-a', making sure 'C-a' works in other tools
bind C-a send-prefix
# Remove original prefix binding
unbind C-b
# Set vi mode
setw -g mode-keys vi
# Move to different windows using vim keybindings (hjkl) instead of arrow keys
bind-key h select-pane -L
bind-key j select-pane -D
bind-key k select-pane -U
bind-key l select-pane -R
# Name windows after directory
set -g window-status-format '#I:#(pwd="#{pane_current_path}"; echo ${pwd####*/} | cut -c1-20)#F'
set -g window-status-current-format '#I:#(pwd="#{pane_current_path}"; echo ${pwd####*/} | cut -c1-20)#F'
# Open new windows and panes in same directory as active
bind c new-window -c "#{pane_current_path}"
bind '"' split-window -c "#{pane_current_path}"
bind % split-window -h -c "#{pane_current_path}"
I didn't think my tmux-configuration was that important, but when you work mostly in the terminal, it's important to be able to navigate with ease.
My setup has reduced drastically for vim. Going from ~500 lines, 7 vimscripts, and 3 plugins, I'm now down to 9 lines and a single script:
set expandtab " Use spaces instead of tabs
set tabstop=4 " Number of spaces a tab counts for
set shiftwidth=4 " Number of spaces to use for each level of indentation
set softtabstop=4 " Number of spaces that a tab counts for while editing
set autoindent " Keep the same indentation as the current line
set smartindent " Automatically add extra indent in certain contexts (like after `{` in C-like languages)
set number " Show line numbers
filetype indent plugin on " determine file type and use auto-indenting
source ~/.vim/vimscripts/commentary.vim " Comment/uncomment with gc[motion]. Credit: Tim Pope <http://tpo.pe/>
I currently don't feel the need to add any plugins to vim.
Previously I used fzf2 quite a lot, both in bash and vim. It is an awesome tool, and it felt like I supercharged my efficiency in the terminal when I started using it.
Now, I realize that the gains were not as large as I thought, and that can do my work efficiently without fuzzy search.
I can easily navigate and find what I need without relying on a complex integration of fzf, bash, vim, find, bat, grep, and ripgrep, which is what I had before (and spent a long time on setting up).
Maybe one day I'll be back there, but not until I feel the need for it.
All in all, I love how these tools are designed to be easily configured, adapted, and integrated with eachother. It's how all software should be designed: Giving people power over the tools they use.
But I also think it's important to step back once in a while and think about how we use the tools, and what tools we actually need.
It is a good reminder to observe that what I once thought of as a large improvement, actually did not bring as much benefit as I thought. Maybe that applies to other things in life as well.
Technically a terminal emulator. ↩
In my job I write a lot of technical text, whether it is for reports or scientific papers. As many others, I have been experimenting with how ChatGPT or other AI tools can be of use for so-called "knowledge workers".
I have noticed two particular things with the text produced by ChatGPT that annoys me:
There's absolutely nothing wrong with these words in themselves, but the way they are overused by ChatGPT makes the text sound very unnatural.
I have, however, a greater concern regarding how ChatGPT modifies and changes the text we use to communicate information. Using an LLM as a tool for expressing yourself means that it's no longer your voice, or at the very best, it's a modified version of it.
LLMs are making their way everywhere, with ChatGPT being only the beginning of AI-access for the greater masses. Apple recently revealed Apple Intelligence1, which is their new AI platform that is "built into your iPhone, iPad, and Mac to help you write, express yourself, and get things done effortlessly." Microsoft, Google, and Meta have already deployed some AI tools that are deployed into their platforms, at least in some regions, but I see Apple Intelligence as the next big step of AI coming into the daily lives of larger groups of people.
ChatGPT has attracted a lot of users that have an explicit interest in using AI-tools, but platform integrations will bring AI also to those that do not necessarily feel the need or want for it. What consequences will this have? Apple claims that their new AI platform will "help you write" and "express yourself"1, but how much are you really expressing yourself with an AI tool as the middle-man?
LLMs has a myriad of use cases, some good and some bad. In my opinion, after one and a half year of observing and experimenting with the chatbot since its release, I think we should be careful about using it as an intermediate layer of personal communication between people. It is not a matter of whether AI will be "good enough" at mimicking our personal style or not. I think there is irreplacable value in the fact that the sender expresses their own thoughts. Receiving AI-generated communication is completely different from something written by the sender on their own, as has been noted by others2.
The landscape of textual communication is changing, and I'm not sure how well my thoughts on this will hold up five years from now. Maybe the expectations for digital communication will change significantly along with the development of AI tools, rendering these concerns outdated. But with so much communication already happening online instead of in the real world, I'm very concerned about how this will affect our inter-personal relationships. Of course, it's a huge difference between, let's say, an SMS and a technical report, and the demands for productivity in the workplace may make it impossible to avoid using AI-tools to speed up communication.
However, I know that when I have a discourse with another human, I prefer reading or hearing the actual words chosen by the other person, because there is also a lot of information in those choices. I prefer hearing your voice, not the one suggested by ChatGPT.
Another collection of interesting reads from around the web.
I was fortunate enough to get the opportunity for a travel to Japan for a work-related conference earlier in November. Since I've never been to Japan, and most likely won't go there again, I wanted to stay a few extra days for a small vacation. My main goal was to see some of the Japanese nature, and ideally visit some mountains. I researched the mountain areas of Honshu (the largest island of Japan), which are often referred to as the Japanese Alps, but going there would require spending too much of my precious time on trains and buses. However, Mount Fuji, Japan's highest peak, was not too far from Tokyo, where my flights would arrive and depart. When I only had time to climb one mountain, the tallest one is a natural choice.
While Mount Fuji is visited by large amounts of people during the summer, it's a different story in the so-called off-season. The authorities go so far as to close all the trails, and based on the information I found it is required to submit a form to the police in order to request permission to scale the mountain. Many websites portrayes an off-season climb as extremely dangerous, and disencourages anyone to attempt it.
I downloaded the form, where it is required to provide information about all the members of the expedition, what gear you have, and how much food you are bringing. I felt the chances of being given permission for a one-man "expedition" in a one-day push was very low. It didn't really make sense that they would be this strict for such a non-technical mountain, but I can understand that the authorities want to avoid unnecessary rescue operations. I had a feeling that it should be possible to climb the mountain without too much fuzz. I found a guy through social media who had been to the summit earlier this fall, who confirmed this. He had no problem getting access to the mountain, and the climb was not as demanding as they make it seem. Crampons and ice axe was necessary, but otherwise it should be fine. I packed the suggested ice gear, along with shell clothing, down jacket and an emergency shelter. I was unfamiliar with the weather and climate in the region, and wanted to be prepared for cold temperatures and wind.
When I arrived in Japan, the weather did not look good for the day I had planned to summit. Up until the morning of summit day I had actually decided to skip the climb, and rather focus on seeing more of the Japanese culture. The whole mountain was completely covered in clouds, and I didn't want to spend my very limited time in Japan walking in fog, not seeing anything, and most likely have to turn around before the summit. I started the day with a run past the Arakura Shengen Shrine, and up into the forest-covered mountains around Fujiyoshida, where I was staying. However, the desire to ascend a real mountain became too strong, and I changed my mind. I went back to the hotel, packed my gear, and ran to Kawaguchiko, where I jumped on the bus to Fuji-Subaru 5th Station. The 5th Station is the most common place to start hiking in the summer. According to what I read online, the bus didn't operate past October, but evidently that was wrong.
When we approached the final stop at 2300 masl I realized that the clouds were quite low, and above this altitude the sun was shining. After a brief look in the souvenir shops, I made my way towards the trail, where you have to pass a large barrier with signs informing you that the trail is closed for the season. Another barrier is further down the trail, just before the path starts ascending up the mountain.
Signs indicating the closing of trails for the off-season. The orange barrier can be seen in the background.
The first part of the trail is what I would call a volcanic gravel road. After a while it gets more rocky, and you're walking on a proper "trail". The path is fenced by chains and ropes almost the whole way up, so it's virtually impossible to get lost. At about 3000 masl I had to put on crampons. There was not a large amount of snow, but the path and rocks surrounding it was covered in relatively hard ice that would have made it impossible to move upwards without spikes. The thinner air was getting noticeable, and I had to move slower than normal, with my breathing getting heavier.
At the bottom of the trail.
Finally up at the crater rim, I could enjoy a fantastic view of both the crater itself, and the never-ending sea of clouds that surrounded the mountain. At the opposite side of the rim was my final destination: the highest point of the mountain, Kengamine. The wind was quite strong here, so I often had to lean into the wind in order to move forward. As the peakbagger I am, I walked over all the small peaks surrounding the crater on my way to the highest point.
Cloudy weather looks different from above.
A weather station is placed on top of Kengamine, and here the wind was particularly strong. I snapped some photos, and walked a bit down again before I took a break to eat some snacks and enjoy the view. It was a fantastic feeling to experience Mount Fuji towering over the clouds, sun shining, and having the whole mountain for myself (as far as I could see at least, and I saw no other tracks in the snow and ice). And to think that I almost decided not to go!
View of the crater from Kengamine.
I continued to follow the rim back to where I had come up, completing the crater circuit. At one point I fell down a rocky slope, heading towards the top of a small cliff. A sudden, forceful gust of wind pushed me off my feet, and one of my crampons got caught in the ice, preventing my from finding my footing. I tumbled down the icy rocks, going head over heels a couple of times before I manage to stop myself. Luckily I only scraped my knee.
Enjoying the views.
From the crater rim I moved effectively down again, jogging and running. Further down, I met a couple of guys who had had to turn around because they didn’t have crampons. Looking at the watch, I saw that I had time to catch the last bus down from 5th Station, but I decided to go down the complete Yoshida Trail, down to my hotel in Fujiyoshida. I had already spent too much time sitting on buses and trains on this trip, and I wanted to see more of the Japanese nature.
The way down.
At about 2400 masl the trail descends into the forest. The mist lay thick between the trees, and there was an almost eerie atmosphere. There are several ruins of old pilgrim logdings and shrines along the path, with information signs about the rich history of the Fuji mountain.
When the trail passed into the flatlands between the mountain and the city, signs started appearing about bears in the area. I didn't think too much about it, but suddenly I saw movement in the woods ahead of me. About fity meters away, the dark shape of a bear ran straight across my path and disappeared among the trees to my left. I had been running in order to get back before it got dark, but now I made a sudden stop. I felt a slight regret that I didn't buy one of the "bear bells" that every hiker in Japan wears, but I’m not sure how effective the actually are. Slowly, I walked past where the bear had appeared, and talked out loud as a substitue for the bear bell. I hadn't made any effort to find out if these bears are aggresive and how one should begave in the vicinity of one. After a couple of hundred meters I decided that I was safe, and I resumed running towards Fujiyoshida. I concluded my trip at the Kitaguchi Hongu Fuji Sengen Shrine, before I jogged back to the hotel and ate dinner.
Kitaguchi Hongu Fuji Shengen Shrine, the official starting point of the Yoshida Trail.
My trip to Mount Fuji was a wonderful experience, and I feel lucky to have experienced it in sunshine and solitude. To be clear: Climbing Mount Fuji in the off-season requires experience, and being in the mountains always entails risks, especially with regard to weather and changing conditions. Having the correct clothes and gear, knowing how to use it and knowing your own limits is essential. However, the mountain is definitely accessible for experienced hikers when the weather and conditions allow it.
I'm fascinated by the world of personal websites that exists on the Internet, often referred to as the Small Web. It's a stark contrast to the "commerical web" and social media platforms, with endless scroll, advertisements and surveillance. The Small Web let's you browse and discover at your own pace, with fewer distractions and less screaming for attention. Content can speak for itself, and it's not pushed in your face.
The Small Web doesn't have any built-in mode of discovery, which I think it's one of its strengths. One of my favourite ways of discovering other people in the Small Web is by recommendations from others, which is why I think it's useful to have a blogroll (here's mine). In addition to that, there are several websites that serve as good entry points, and I've tried to compile a list of them here. If you have any other good suggestions, let me know; I will update this list consecutively.
Here are some places to discover other people's ideas, thoughts and projects:
Read more about the Small Web here:
The landscape of social media is changing. Twitter is in the middle of a considerable decline, and Meta's Facebook and Instagram are trying to adapt to new requirements and legislation. The decentralized social network Mastodon is gaining serious traction, giving some hope to the dream of social media platforms that are not built on surveillance capitalism. Now that we finally have viable alternatives to Big Tech platforms, is social media salvaged?
I joined Mastodon in late 2019, which was relatively early and before it had significant mainstream attention. I had been interested in alternatives to the corporate-owned platforms for a long time, and wanted an option that was not infested with advertisements and behavioural tracking. In this process I had explored things like Hubzilla, Friendica, Diaspora, OpenBook/Okuna and Scuttlebutt. Each had their own set of advantages and weaknesses, but the ultimate problem was always a lack of critical mass. The platforms based on the ActivityPub protocol seemed more promising, especially since one of them, namely Mastodon, after a while started to get a decent amount of users. However, the community (or rather communities) consisted mostly of tech people, and the larger part of the posts also revolved around tech, from what I could observe at least. Not an easy sell when you want your non-tech friends to join you in the transition to decentralized platforms.
I made a draft for a blog post three years ago called "Trying out the federated social medium Mastodon". I wanted to highlight some of the things I observed, and give the new type of social network some attention. I never published the post. At the time I really wanted to advocate for trying something else, but the amount of activity on Facebook and Instagram was (and is) just too large for most people to miss out on. I had spent some "social credit" on getting people around me to use Signal instead of Facebook Messenger, even though it was only successful for my closest family, and I still have to use Facebook Messenger almost just as much today in order to keep in touch with my friends. I have never been particularly active on social media anyways, so I didn't (and still don't) have much leverage in trying to convince people to try out alternatives. Only people with a large outreach could possibly influence others to join them in switching platforms.
After Twitter started struggling for various reasons, Mastodon has seen a large influx of users. Three years ago, I would have been elated to see that a decentralized social network is finally succeeding, but I'm less excited than I expected. Of course, it's a massive improvement to have platforms that put more power into the hands of individual people rather than Big Tech companies, but the feeling I get is that it's still just another feed to scroll.
We can finally escape the ubiquitous advertisements of other platforms, but I realize more and more that being on social media is usually not worth the time. The true value of social networks is the possibility to have a discourse with anyone, anywhere, but a lot of the time it's more about consuming content rather than communicating. The staggering amount of information that we process while scrolling through our feeds are taking up precious bandwith in our attention, and the discussions we engage in online are often less than constructive. I'm of course aware that people use social media in very different ways, so others' experience may not be as negative as my own. On the other hand, you may also think that I'm stating the painfully obvious.
It's not a controversial opinion to say that meeting and communicating physically in general is more meaningful and valuable. Looking at how social media has changed, and how it as affected my own life, I want to start putting more effort into the interaction I have with people in the "offline" world rather than online. It will most likely entail relationships with fewer people, and at a slower pace, but that's sort of also the point.
For digital communication, I see an advantage in prioritizing one-to-one interactions, rather than the many-to-many communication that modern social platforms enable. I find myself coming back to the actual "mastodon" of social networks: Email. "Everyone" has an email account, and the effort it takes to write an email usually ensures a certain level of commitment to the message. I use RSS as the main way of keeping up with updates to personal blogs, magazines and podcasts. It's a calmer way of consuming. Using personal websites as a way of publishing online also enables more control of both content, form and creativity.
The fact that I have a high threshold for posting on social media hinders a lot of engagement with other people, whether it's communication with friends/acquaintances, academic or work-related discussions, response to blog posts or sharing dog pictures. I'm definitely missing out on opportunities and interesting discussions. Even so, at the moment I feel more inclined to be more present in the moment, as the cliché goes, and in order to do that I have to be less online.
In answer to the question implied by the title of this post, I hope and believe that the pendulum will start to swing back from the "terminally online" culture that social media brought with it. I'm unsure whether it is possible to design large-scale digital tools for social networks that work well for humans, and there are signs that many are fed up with being "connected" all the time. Without discarding the tremendous benefits of connecting the whole world through the Internet, I think it's wise to take a break and assess how we actually are affected by our ever-online lives.
A few interesting reads on the topic of generative AI and large language models:
According to [the logic of Google CEO Eric Schmidt], the failure to “solve” big problems like climate change is due to a deficit of smarts. Never mind that smart people, heavy with PhDs and Nobel prizes, have been telling our governments for decades what needs to happen to get out of this mess: slash our emissions, leave carbon in the ground, tackle the overconsumption of the rich and the underconsumption of the poor because no energy source is free of ecological costs.
Lately, I've been playing around with LLMs to write code. I find that they're great at generating small self-contained snippets. Unfortunately, anything more than that requires a human to evaluate LLM output and come up with suitable follow-up prompts. Most examples of "GPT wrote X" are this - a human serves as a REPL for the LLM, carefully coaxing it to a functional result.
A few weeks ago, a colleague and I wrote an op-ed in Teknisk Ukeblad where we played around with the idea of how to democratize generative artificial intelligence (AI) and avoid ending up with a monopoly on large generative AI models. The possible scenarios we envision may not all be realistic, but we want to highlight the possibilities we have to make the use and development of AI more transparent.
The op-ed is in Norwegian, but could be translated to English using various translation tools. Read the op-ed here: ChatGPT requires moderation like Wikipedia (the original title was Democratizing Generative AI, which we feel is a bit more representative of what we wanted to convey).
One year ago I wrote a blog post about machine learning and carbon emissions. Since then, we have experienced some ground-breaking developments in the world of AI. With the release of image generators like DALL-E and large language models (LLMs) like ChatGPT, we are talking about a whole new scale of computational load for using, let alone training such models. What's more, these are no longer tools reserved for research purposes, but deployed software that people all over the world has started using on a daily basis.
In my research group, Trustworthy Green IoT Software at SINTEF Digital, we are exploring how we can contribute to Green AI, which refers to environmentally friendly AI development. One of the tools we have developed, an ML pipeline for supervised learning, uses DVC for pipeline orchestration, which we see as a measure of reducing the energy usage through caching and skipping unnecessary reruns of pipeline stages. Many researchers are calling for measuring and reporting the carbon footprint in order to bring awareness to the issue, which is something we want to bring into our existing tools. The question is whether such measures are sufficient when the power of LLMs and similar tools is pushing the development in a much more resource-intensive direction.
If you are an AI/ML researcher or practitioner, how do you look at the increasing carbon footprint of ML models, and do you see "Green AI" as an important aspect for the future of AI development? Will the power of extremely large models always win over models with smaller footprint, or is it possible to find a balance between performance and energy efficiency?
One of my new year's resolutions: Always ski a new line through the forest when going back to the cabin. It surprised me how much fun I can have in my own backyard just by exploring a bit more.
This is the year of glade skiing for me.
A fun afternoon skiing with my Border collie Gjert in our backyard.
I have a large collection of plain text files (mostly Markdown) that contain all my notes and writings, both for work and non-work, which I navigate and modify using Vim, fzf, shell scripts and command line utilities (on mobile I use the excellent app Working Copy). Everything is of course synced and version-controlled using Git.
I recently discovered that Obsidian has a range of interesting features, and since it is meant to manage a collection of Markdown-notes (which still can be synced through a git-repo between desktop and mobile apps), I could try it out without having to do any changes at all to my current structure or formatting of my files. I downloaded the apps for desktop and mobile, and everything was good to go.
The most exciting features for me in Obsidian are the following:
I think Obsidian works great in general, and even though it is closed source software, anyone can write plugins for it. There are many useful "community plugins" that really increases the usefulness of the program. However, I missed writing in my usual text editor at the command line.
Obsidian has the option of using Vim keybindings, but that does not replace the configurability of Vim, and the advantage of using command line utilities and scripting as a part of the workflow.
Also, I kept hitting ctrl-W [h,j,k,l] in order to change between panes in Obsidian, but of course ctrl-W means "close file" when you are outside the command line.
The user experience of Obsidian on iOS is a bit "messy" in my opinion, and I found myself opening Working Copy more often than Obsidian to edit files. Navigating the file explorer in Obsidian on both desktop and mobile is less than optimal, among other things because of no easy way to collapse all folders, and I found myself scrolling a lot.
Some of my complaints can be fixed through plugins, either existing ones or by writing one myself, but the power of the command line is either way out of reach. Because if this, I wanted instead to reconstruct some of the Obsidian features I liked the most in Vim (or at the command line)
To make this work, the links to other files has to be relative paths (otherwise images and links won't work properly when rendering the Markdown in HTML).
With the plugin fzf.vim, the following line maps ctrl-x ctrl-f to search for any file in the parent directory from which you opened Vim (which means I always open Vim from the root directory of my note collection) and insert its relative path from the current open file when you press Enter:
inoremap <expr> <c-x><c-f> fzf#vim#complete("find -print0 <Bar> xargs --null realpath --relative-to " . expand("%:h"))
The line above is put in your .vimrc (Vim configuration file).
Made based on tips from this GitHub-thread.
Vim already had functionality for this, I just didn't know about it.
ctrl-w ctrl-f opens the filepath under the cursor.
This one was easily implemented in a bash script:
#!/bin/bash
file=$(find /path/to/markdown-collection/ -name "*.md" | sort -R | head -1)
vim $file
The plugin fzf.vim makes navigation very easy, both inside Vim and at the command line.
When having files open in Vim, the command :Files let's you fuzzy search for filenames, and :Rg (if you have installed ripgrep let's you fuzzy search for file contents.
I'm quite restrictive of adding specific configuration and plugins to my Vim setup and want to keep things lean (I only use two plugins currently), but fzf.vim has really made my workflow much better.
I use linking between files quite a lot, especially for embedding images into Markdown-files. Sometimes I restructure or rename filenames, and since I might have embedded one image into multiple files, it would be a massive time saver if all the links could update automatically when moving or renaming a file. I tried searching for an existing solution; the only similar thing I could find was an extension for VSCode, but no solution for Vim.
I have made an attempt to make a shell script for moving/renaming files, which automatically updates all links to the file in Markdown-files of the working directory. You can find the current version of the script here: linkmv. I'm pretty inexperienced in bash, and I haven't tested all the possible edge cases, but it's a work in progress. Hopefully I'll get something working eventually (contributions and tips welcome)!
Sharing some links I find interesting about note-taking:
The dilemma of digital versus analog tools, especially related to note-taking, is something I always return to. Even though digital tools typically are the first choice in many situations, we shouldn't forget the strengths of a simple pen and paper.
I usually discover new books to read through references or quotes from previous books I have read, but sometimes I also get some useful recommendations from blogs or people I talk to. Maybe I can provide some tips on your next read?
Here's a summary of some of the best books I read in 2022:
This book gives a very important perspective on how we may solve the challenges we face regarding climate change and over-consumption of resources.
Some innovations, and perhaps those to be feared most because they are often ones that are profitable, allows the absurd needs generated by systems to be fulfilled. Nanoscale particles allow a sunscreen to offer protection and a feeling of freshness; RFID chips makes it possible to pay for the contents of your supermarket trolley without going to the cash register; office glazing can be made opaque by eletrical impulse; refrigerators prepare the shopping list. We are far from the "revolutions" of the steam engine and crop rotation! These examples are caricatures, but emblematic of many of today's "innovations". Frankly, would we develop such high-tech applications if we knew the real price to pay, in terms of factory waste, the disappearance of nature and resources, the destruction of everything that has sustained us, physically and intellectually, for millenia? We cannot say.
Very interesting tales from "the golden age of mountaineering" (1850s and 1860s) when many major first ascents in the Alps were achieved. Edward Whymper was one of the pioneers of mountaineering, and is most famous for his first ascent of the Matterhorn, which is described in this book. It is wise to remember that this book only provides Whymper's account of the controversy on Matterhorn, which has been disputed by other members of the first ascent party.
Whymper seems to be a typically "manly" man, who enjoys the harsh challenges and dangers of high-altitude mountaineering. The following quote gives an impression of how he thinks about risks in the mountains:
The uppermost men were continually abused for dislodging rocks and for harpooning those below with their bâtons. However, without these incidents the climbing would have been dull: they helped to break the monotony.
Even though the main focus of the tales is Whymper's accounts of all his glorious first ascents, the book also gives some perspective on what we learn from mountaineering:
I have not made myself an advocate or an apologist for mountaineering, nor do I now intend to usurp the functions of a moralist, but my task would have been ill performed if it had been concluded without one reference to the more serious lessons of the mountaineer. We glory in the physical regeneration which is the product of our exertions; we exult over the grandeur of the scenes that are brought before our eyes, the splendors of sunrise and sunset, and the beauties of hill, dale, lake, wood and waterfall; but we value more highly the development of manliness, and the evolution, under combat with difficulties, of those noble qualities of human nature--courage, patience, endurance and fortitude.
For those who have an interest in mountaineering, this is a must-read.
This is another essential read for those who love mountains and mountaineering. Bonatti writes a lot about how climbing affects the human mind, and it is very inspiring to read about his principles and his approach to mountaineering. Similar to Edward Whymper, Bonatti was involved in the controversy of a first ascent, in this case of K2, one of the most difficult mountains to climb in the world. I recommend reading the version that is co-authored with Robert Marshall, who has gone through the evidence thoroughly and gives perhaps the most accurate account of what actually happened on K2 during the first ascent.
Bonatti also provides his thoughts on what mountaineering really means for us:
What is there, beyond the mountain, if not the man? Mountaineering is only one of a thousand ways of living and getting to know yourself. Climbing mountains should signify nothing more than this search for identity. It should never be mere escapism, because sooner or later we must return to our own personalities and feelings, the only place available apart from nothingness. So the mountain should prepare us to go further. A mountaineer should store experiences and use them to enrich himself. He can do so if he knows about wide open spaces and taking responsibility for his own actions.
One of my all time favourites: A book about a man's real life experience with living without technology. For those who long for a simpler and more meaningful life, this will give some inspiration.
This book was written in the 1970s and outlines problems related to media and specifically television, but it is just as relevant today, and most of the reasoning can be applied directly to how we are affected by social media and technology today.
Strava is a popular tool for sharing sports and outdoors activities, and has become the de facto social medium for athletes, runners, cyclists etc. I'm not the kind of person who likes to share every detail about how I train or my daily activities, but sometimes I enjoy sharing a workout, race or a route I've done in the mountains. I also enjoy keeping control of my own data and how it is accessed and displayed, and I have only reluctantly used Strava and Garmin Connect to share activities.
While I'm waiting for someone to make an ActivityPub-enabled Strava-clone, I made my own Activities-feed on this website. To add an activity, I upload a gpx-file containing the GPS-track (including elevation data and timestamps), and optionally add a textfile with a description and images from the activity. The feed uses Leaflet and leaflet-evelation.js to display gpx-files on map tiles, and plot the elevation profile and speed graph. I have chosen to use map tiles from OpenTopoMap, since it's important for me to have detailed terrain information when visualizing the activities. My very messy build-script for this website takes care of putting the text description and images in the correct places.
This way I can easily share workouts and activities with friends and family. The activities also show up in my RSS-feed. The feed is not very visually pleasing yet, but I will probably fine-tune to CSS to make it look a bit more clean.
Check out the new feed at this link!
I shot a short video of Gjert the border collie during one of his agility workouts. I am using the GoPro HERO7 Black, with no gimbal, and I must say I am impressed with the HyperSmooth video stabilisation.
Take a look!
There are several very exciting projects that are aiming to make more sustainable consumer electronics:
Such projects have inspired me to put together my own "modular" laptop, with parts that are easily replaced. I'm very fond of the Raspberry Pi computers (henceforth referred to as RPi, RPis plural), which are cheap single-board computers that are made to be tinkered with. One of my RPis is the RPi 4, with 8GB RAM, which is the most powerful RPi available to date. I wanted to use this one to make a laptop with the following requirements:
In addition to the RPi itself, I needed the following parts:
For the display, I ended up with the official Raspberry Pi Touch Display 7". The size was about what I had in mind for a portable computer, and I was also lucky enough to find someone who sold a used one for a cheaper price.
I also found someone who sold a used Logitech K400, which is a keyboard with an integrated touch pad. I didn't really want a separate mouse for my RPi laptop, so this was perfect.
For portable power supply, I just use plain powerbanks. I know that one should preferably use a something that reliably supplies the exact voltage that the RPi needs, but I find that powerbanks works just fine in most cases. For some very small powerbanks, I somtetimes get the "low voltage" warning on the RPi, but apart from that, it works flawlessly.
The hardest part was binding all the parts together in a package that can easily be carried around. I didn't want to have several separate parts that I need to pick up in order to move the laptop. After a lot of experimenting, I found a adequate solution by using adhesive GoPro mounts and some extra arm joints. I put one adhesive mount on the keyboard, and one on the display case. I connected these by using arm joints (one of them needs to be slightly modified by cutting of one of the plates). The powerbank can be fastened to the "display foot" using an elastic band or similar, or to the keyboard.
The result is a portable computer (I call it a laptop, since it's a computer I can have on my lap, but I'm not sure if everyone agrees with me), with parts that are very easy to get hold off.
I cut off a part of the display case in order to easily swap memory cards (which I do frequently, since I like to use different types of Linux operating systems).
A small part of the display case is cut off, in order to easily access the
memory card that the OS is stored on.
The laptop can easily be disassembled into these parts.
As a bonus, the computer can be mounted on other things that have a GoPro mount on them. I have a strong clip with a mount on it, which enables me to attach the computer at the edge of a table or similar. This is very useful when I want to watch the display when I'm cooking or washing the dishes.
The computer and display mounted on a clip instead of the keyboard. Saves
desktop space.
The RPi isn't the most powerful computer in the world, but it can handle a lot of day-to-day stuff, like text editing, internet browsing, watching videos, playing games, listening to music etc. I have used mine a lot, and one of the best things with it is that I don't have to be afraid of breaking it, since I know that all the parts, even the main board itself, is quite cheap to replace.
I prefer having control of my own data, and therefore try to avoid using cloud services if I absolutely don't have to (e. g., at work I'm pretty much forced to use the cloud services of my employer, which is Microsoft of course). Earlier I have been self-hosting some services, including file storage (Nextcloud), music server (Subsonic), git (gitea) and social media (Hubzilla). Then I moved to a place where I couldn't easily set up my own server with port forwarding for external access, and my self-hosting was put on pause. As an experiment, partly motivated by the increased focus on the environmental impact of the expanding use of servers and cloud services, I attempted to reduce my use of cloud services to a minimum. Some of the things I did:
I moved to a more offline workflow, and realized that I don't need to have all my digital stuff instantly available on any device with an Internet connection. This also have the added benefit of enabling me to go offline, which often improves my focus, without losing access to my files. Since I now live a place without Wi-Fi, it is a great advantage that I'm not that dependent on bandwith.
The single most important cloud service for me, both professionally and privately, is git hosting. It is where I sync most of my work, code, notes and writings. Some things are private/personal (or at least should be hidden from the eyes of others), which of course are kept in private repos, but I would much prefer to host those repos myself. One option I have been considering is to pay for hosting of my own gitea instance, but I both want to save money and avoid having another cloud service (even though it would be managed by myself). I could also choose to have local repos without storage in the cloud, but I need to easily keep the repos synced between my personal latop, work laptop and phone. I started playing with the idea of having a portable git server on a Raspberry Pi Zero, powered by a power bank, which could set up its own Wi-Fi and start a git server automatically when booted and let me push and pull from repos hosted on it. This would however require me to carry an extra device.
I then realized that maybe my phone, an iPhone, could serve as the git server instead. The App Store lists some apps that claim to be able to setup a git server, but after reading a couple of the prvacy policies, I decided I would sleep better at night if I didn't have to trust some ambiguous wording on what and how data is collected. I use the Working Copy git client app a lot, which can serve the git repos in the app on a WebDAV server. I was able to access the server through Finder on macOS, but I gave up after failing to connect to it at the command line from Ubuntu. Instead I tried setting up a git server at the command line in the iSH app. It proved to be the simplest and most usable approach for me. When using the iSH app, the setup is identical to how you would do it when setting up a git server on a normal Linux machine. In the app I installed git and git-daemon (they have to be installed individually):
apk add git
apk add git-daemon
Then I initialized a bare git repo:
mkdir myrepo.git
cd myrepo.git
git init --bare
To serve the repo (and any other git repos in the same folder:
cd ..
git daemon --reuseaddr --base-path=. --export-all --verbose --informative-errors --enable=receive-pack
If you want to disable the ability to push to the repos (i. e., clone-only access), skip the last flag.
To clone repos into Working Copy on the same phone, use git://127.0.0.1:9418/myrepo.git or git://localhost:9418/myrepo.git as the URL (9418 is the default port of git daemon).
I don't think iSH handles running inthe background well, so you might have to press "Clone" in Working Copy (after filling out the URL), switch to iSH to enable the server to complete the request, and then back to Working Copy.
To clone to other devices, run
git clone git://[ipaddress]:9418/myrepo.git
where [ipaddress] is the IP-address of your iPhone on the local network (the other devices obviously have to be connected to the same network).
The drawback is of course having to start the server before every push/pull, but for my uses this is acceptable. Also, there is no authentication required to pull from the server, so one needs to be sure that there are no bad actors on the local network.
Next up is trying the same thing on an Android phone (which I expect to be just as easy to do inside Termux), and then maybe setting up a mobile git server with authentication.
Some links I find worth sharing:
In this post I want to share some of my best reads in 2021 (I know we are almost halfway into 2022, but I wrote this post 5 months ago and just now realized I forgot to publish it):
A very well written and thoroughly researched book on surveillance capitalism. Zuboff very succinctly articulates the problems that arise from the (mis)use of modern technology and how corporations gather and use personal data. I highly recommend this book.
Timeless wisdom.
A classic from one of Norway's most celebrated adventurers of the 20th century. Truly fascinating and humorous to read the tales from his time as a trapper in the wilderness in Canada.
The fourth book in one of my favourite fantasy series.
A very captivating account of the tragic story of one of the worst accidents on Mount Everest.
Another classic from Krakauer, and one of my all time favourites in the outdoors genre.
A very interesting take on how modern technology and so-called late capitalism affects our lives.
The last years I have been moving towards plain text files for almost any use case where I need to store information digitally, and after reading about Calendar.txt I have started to use a plain text file as my calendar. I use (almost) the same template as described in the link, but I wrote my own script for generating the calendar template so I can easily make adjustments if needed. Example of the format with a few example events:
2023-01-02 w1
2023-01-02 w1 Mon
2023-01-03 w1 Tue 12:00 Meeting.
2023-01-04 w1 Wed
2023-01-05 w1 Thu 18:00 Going for a run.
2023-01-06 w1 Fri
2023-01-07 w1 Sat
2023-01-08 w1 Sun
2023-01-09 w2
2023-01-09 w2 Mon
2023-01-10 w2 Tue
2023-01-11 w2 Wed
2023-01-12 w2 Thu Seminar.
2023-01-13 w2 Fri
2023-01-14 w2 Sat
2023-01-15 w2 Sun
I find it quite nice since it integrates well with the rest of my very plain text-oriented workflow, and it's easily synced and managed with git together with the rest of my plain text files (notes, todos, etc). It takes some time to adapt to since the "user interface" is quite different than normal calendar applications, but for those who are familiar with quick navigation in text files (for example using Vim), it can be much more efficient to use a plain text calendar.
I also wrote a small bash script for opening the calendar on the current date in Vim:
today=`date +%Y-%m-%d`
vim /path/to/calendar.txt -c "/$today"
Jeg fikk for et halvt år siden en turtleneck fra Northern Playground, et veldig spennende norsk klesmerke. Kvaliteten og bruksverdien til dette plagget var overraskende gode, og hvis du ikke gidder lese hele testen så kan jeg oppsummere den allerede her: Om jeg bare skulle hatt én trøye resten av livet så er det definitivt denne; den fungerer til alt fra trening og tur til jobb og selskap. Northern Playground er det første og eneste klesmerke jeg har vært borti som faktisk leverer alt de lover i reklamene sine, og det er ikke lite.
Det er mange klesmerker som lager klær i ull og ullblandinger som påstår at klærne ikke får noen svettelukt så lenge man lufter dem. Jeg har testet mange ulike merker, men trøya fra Northern Playground er den eneste jeg har prøvd hvor dette faktisk stemmer. I en av sine reklamefilmer forteller Northern Playgrounds CEO at han ikke husker sist han vasket trøya han har på, og at han fortsatt ikke får noen klager på lukt. Dette synes jeg i utgangspunktet det var vanskelig å tro på, helt til jeg fikk prøvd trøya selv. Jeg har ennå ikke vasket min Northern Playground-trøye siden jeg fikk den for et halvt år siden. Den lukter ingenting, til tross for å ha vært med på løpeturer og på jobb uke etter uke. Om jeg har svettet mye i den så henger jeg den til lufting enten på verandaen eller over en stol innendørs, og da er den klar til bruk dagen etter. En annen stor fordel med trøyene er at de er såpass fine at man fint kan bruke dem både på jobb og i sammenhenger hvor man må se litt ordentlig ut.
Etter mye bruk er det kun ett "svakt punkt" jeg har klart å finne i denne trøya. Jeg har løpt mye med sekk til og fra jobb, uten noe plagg mellom sekken og trøya, og jeg ser at det har kommet noen nupper på ryggen på grunn av dette. De er så små at jeg ikke ville tenkt over det med en annen trøye, men når klesplagget leverer såpass godt på alle andre punkter, så må man fram med små tingene.
Jeg har også skaffet meg den kortermede versjonen, som selvfølgelig fungerer like bra, og trøyene er med på løping, klatring/buldring, jobb, selskap og fjellturer. Northern Playground lager trøyer i mange forskjellige fasonger (T-skjorter, langermet, bestefartrøye, zipneck, hoodie etc), så her er det mulig å finne noe for enhver smak.
Selskapet har også en svært miljøvennlig profil. De har flere tiltak for å redusere forbruk, for eksempel at de tilbyr livstidsservice på plaggene, og de har en selvpålagt miljøskatt på hvert salg.
Et veldig spennende prosjekt de har gående er The Last jakke og The Last bukse. Målet er å lage en jakke og en bukse som er de siste du noen gang trenger å kjøpe, og om mulig også gå i arv! Jeg håper de lykkes med å lage et godt produkt, og det har jeg også troa på at de klarer. Northern Playground har en forbilledlig designfilosofi og miljøprofil, og jeg håper de kan inspirere andre klesprodusenter i samme retning.
Bilder av trøya:
Last year I inherited an analouge camera from my grandfather. Usually I prefer digital tools, because they are space-efficient, easy to carry around, and the data they produce can is easy to store and backup. However, there are also some disadvantages. My photo library has been steadily growing each year, which recently resulted in me having to buy another external hard drive in order to have enough storage space for backups. Digital photography makes it possible to take a lot of pictures, and sometimes (or most times?) I'm taking too many. Analogue photography forces you to be more thoughtful, both because of a very limited amount of photos per film roll, and the economical cost of each photo. In addition, I think it's awesome to use a piece of technology that's 50 years old and still works flawlessly.
Here's a few photos from the first roll I shot with the camera:
Send me an email if you think you can recognize this mountain, and I will tell you if you're right (hint: It's in Trollheimen, and not far away from the hut shown above).
Recently I have been looking into the environmental impact of machine learning in relation to ongoing research projects at work. Carbon emissions are high on the agenda for everyone, especially if one works at a research group with the term "Green" in it's name. Lots is already written on the topic, but I wanted to share a few of the most interesting articles and research papers I found, with a few comments of my own.
Deep learning's carbon emissions problem from Forbes summarizes well some of the most problematic issues of machine learning. It is mainly focused on deep learning and natural language processing (NLP), which entails developing models that consist of an enormous amounts of parameters. Many of the article's key points stems from a paper that in 2019 really brought awareness to the gigantic energy usage of some of the largest deep learning models: Energy and Policy Considerations for Deep Learning in NLP. As a machine learning practitioner, the following quote from the article is something I'm especially concerned with:
Another factor driving AI’s massive energy draw is the extensive experimentation and tuning required to develop a model. Machine learning today remains largely an exercise in trial and error. Practitioners will often build hundreds of versions of a given model during training, experimenting with different neural architectures and hyperparameters before identifying an optimal design.
Luckily, there exist several frameworks that can help reducing the number of experiments needed during model development. Personally, I use Data Version Control (DVC), an excellent tool for structuring machine learning experiments. Not only does it helps you keep track of models and hyperparameters, but you can also use it to track all stages of a machine learning pipeline (which typically consists of several preprocessing stages). Since it keeps all intermediate results between stages in cache, you can save a lot of computation time when experimenting with different control parameters.
I think the article concludes with an important note:
The AI community must begin to work toward new paradigms in artificial intelligence that do not require exponentially growing datasets nor outrageously vast energy expenditures.
Another article, The drive to make machine learning greener from Discover Magazine, quotes a researcher team from the University of California, Berkeley, on how reporting on carbon emissions might help reduce them:
Patterson and colleagues conclude by recommending that computer scientists report the energy their calculations consume and the carbon footprint associated with this, along with the time and number of processors involved. Their idea is to make it possible to directly compare computing practices and to reward the most efficient. “If the machine learning community working on computationally intensive models starts competing on training quality and carbon footprint rather than on accuracy alone, the most efficient data centers and hardware might see the highest demand,” they say.
The paper Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning argues similarly for the importance of measuring and reporting the carbon footprint of machine learning experiments. The paper is an especially interesting read, as the authors suggest a set of concrete courses of actions based on their findings. The main points are to be consistent in reporting energy and carbon metrics, and using leaderboards of these metrics as incentive for reducing carbon emissions in research. They also write that one should consider "energy-performance trade-offs before deploying energy hungry models". It's also interesting to note that one of the most intuitive (at least in my opinion) ways of measuring the energy-efficiency of any computer program, namely to count the number of Floating Point Operations (FPOs), actually can be misleading (read section 5.1 of the paper for a discussion on this). This is one of the reasons for why there is a need for frameworks that can measure or more accurately estimate the actual carbon emission of training or running a machine learning model. The authors have published an open source Experiment impact tracker that can be used for this purpose.
Another paper that looks into methods for tracking energy-efficiency is this: Quantifying the Carbon Emissions of Machine Learning. The authors have contributed to two tools that can be employed for emissions tracking. The first one is an emission calculator, where you compute emissions based on hardware type, runtime and cloud provider/region. The second one, which I personally have started testing in my projects, is the CodeCarbon framework. CodeCarbon can be embedded in Python code, and automatically tracks the energy usage and carbon emissions of your experiments. The tool is very easy to use and generates results in a csv-file that can be used for your own analysis. Additionally, the tool provides an interactive dashboard based on the results, which makes it very quick and easy to get up and running with emission tracking.
Even though the emissions of "small-scale" machine learning practitioners like myself are dwarfed by the gargantuan carbon footprint of big tech companies like Google and Amazon, I think it is important that we are all aware of these issues, and try to mitigate them at any scale. A shift towards more green machine learning would benefit us all, and hopefully also affect the large players in the industry.
To summarize, here's a few concrete actions that can help reduce the carbon footprint of machine learning:
Here's some articles I found interesting enough to share this week:
When you're crafting your own website without any website builder or tools like Wordpress, it can be a bit challenging to know how to optimize images. JPG-files from a digital camera is usually a few MBs in size. For example, an image from my Gopro, with a resolution of 4032x3024, generally amounts to at least 2 MB. If you're uploading such files, you're wasting bandwith, since most screens and monitors won't show the images in their full resolution anyway. I have found it suprisingly difficult to find "best practices" for optimizing images for the web using simple tools, but after a bit of research, I've managed to put together a script that combines three different command line tools to reduce image size without loosing to much quality.
The three command line tools are:
exiftool: Edits EXIF-information. Used to remove metadata from the image, such as geolocation, which I don't want to share on the web.jpegoptim: Optimizes JPG-files.imagemagick: Used to reduce resolution of image.Here's the bash script, which takes files as input:
#!/bin/bash
for i in "$@"; do
# Reduce all metadata, but keep orientation of image:
exiftool -all:all= -tagsFromFile @ -exif:Orientation "$i"
# Optimize image to a size of 500 kB:
jpegoptim --size=500k "$i"
# Reduce resolution so the width is maximum 1500 pixels
# (mogrify is a part of imagemagick)
mogrify -resize 1500 "$i"
done
I am by no means an expert on this, but I've found a routine that works fine. If you have any comments or better solutions, let me know.
Yesterday I presented and defended my master thesis in "Computational Science: Physics" at the Physics Department at the University of Oslo. The title of my thesis is Deep learning to estimate power output from breathing.
The main goal of the project was to use recordings of breathing patterns to estimate how much power (measured in watts) a person generates during cycling. The motivation was to investigate whether breathing data can provide a universal and portable way of estimating physical effort/exercise intensity.
Popular methods of measuring exercise intensity include heart rate, speed measurements and power meters. Let me why we want to investigate an alternative to these traditional ways of effort gauging:
Because of this, we wanted to see whether breathing data could be used to estimate how much power a person outputs during exercise! So to summarize, the main goal of the project was to find a relationship between these parameters:
The breathing data consisted of measurements of how the ribcage and abdomen expands and contracts (this type of measurement is called respiratory inductive plethysmography, or RIP). I used two of these sensors, as shown in the picture below.
Image of the RIP sensor to the left, and a figure showing how a person would
wear two of the sensors the to right.
In order to find the relationship between breathing data and power output, I used neural networks, which is a type of machine learning.
I collected data from myself (it's fantastic to take a workout and be able to say "I'm just working on my thesis"), by recording RIP data, heart rate and power output during a series of workouts.
The data flow during data collection.
The RIP sensors contains a strain gauge that converts the mechanical strain on the belt into an electric signal, which means that the sensor outputs values measured in millivolt. An example is shown below, and it's easy to see how the signal increases when the rib cage expands, and decreases when it contracts.
Raw RIP data from the sensor around the ribcage.
Even though it's easy to recognize the breathing pattern when looking at the graph of the RIP signal above, it's not easy to discern how it behaves with regards to exercise intensity, at least not by visual inspection. Below is a graph showing an excerpt from an interval workout. The two RIP signals (in the two top graphs) have some strange behaviour and seem to jump around a bit. This can be a result of belt slippage, or that the person for example is standing up while cycling, reaching for a water bottle, or similar. Hopefully the neural networks can help us see past such irrelevant information in the input data!
Example of raw data from an interval workout.
I performed many different types of workouts, various types of intervals, steady-state efforts and workouts with ramp and pyramid structure. Some examples are shown below.
The data set consisted of workouts from mainly three different categories.
In order to enable the neural networks to make the best possible estimations, I needed to extract/engineer/derive (choose the term you feel is most understandable) some features from the raw data. To put it simply: Feature extraction means that I'm trying to find the most important information in the raw data and feed that to the neural network, so it's easier for the neural network to learn the relationship between the input and output data. But this also means that I have to figure out what the most important information is, and in the end, that's what I want the neural network to do... So you could say that it's a sort of collaboration between me and the neural network!
I extracted many types of features, but three of the most important ones are shown in the figure below. To keep this blog post short and sweet, I won't go into the details here.
This graph visualizes three of the most important features that a extracted
from the raw data. Read the thesis (link at the end of the post) for more juicy
details!
My best performing model was obtained by using a convolutional neural network, and when testing the model on unseen data it got an average error of 20%. That may sound like a lot, but the average error is affected by the fact that the model is not very good at estimating power values above 400 watts, since most of the data set contained power values below that (which is also where most of us spend most of the time when cycling).
An example of how the power output estimation by the model looks like is shown below. The output signal (in blue) is quite noisy, but the model is able to estimate approximately the correct power level most of the time.
Example of power output estimation, using extracted features from the raw
data (the blue line shows the "predicted" values, which in this case is
analogous to "estimated".
As a comparison, take a look at how the model performs when using only the raw data, without any extracted features (spoiler alert: Not well at all)!
To emphasize how important feature extraction is, here is an example of power
output estimation using only raw data.
Conculsion:
If you have some questions or thoughts about my thesis, or about the subject in general, please send me an e-mail on erikjohannes@protonmail.com!
Here are some links to learn more about the research project:
For litt siden fikk jeg muligheten til å være med å teste The North Face Flight Vectiv, som er en ny løpesko fra The North Face med en innebygd karbonfiberplate (siden det funka så bra for Nike og for gateløping, så må det jo prøves i terrenget også). Jeg har testa disse skoene noen uker på skogsstier, snø, grusvei, i myr og i lyng rundt om i nærområdet, både i vått og tørt vær.
Førsteinntrykket er at disse skoene er skikkelig høye, og jeg synes de er ustabile på ujevne og tekniske stier. Jeg løper vanligvis med litt lavere sko i terrenget, Arc'teryx Norvan SL (høyde: 19/12 mm på hæl/forfot) og Salomon S/LAB Sense 8 SG (høyde: 22/18 mm på hæl/forfot). Jeg var sikker på at Flight Vectiv hadde en mye høyere mellomsåle en dette, men da jeg sjekket opp spesifikasjonene på nettet så fant jeg ut at de en oppgitt høyde på bare 25/19 mm! Altså er det ikke så stor forskjell, selv om jeg opplevde at Flight Vectiv-skoene var mye høyere enn det jeg normalt bruker. Arc'teryx- og Salomon-skoene mine har ikke noen innersåle, men det har Flight Vectiv, så jeg begynte å lure på om den oppgitte sålehøyden kanskje ikke er målt med innersåle..? Uansett, det endrer ikke på at jeg opplevde Flight Vectiv-skoene som uegna for tekniske stier. Dempingen gjør forøvrig at de er greie å løpe med i nedoverbakker på jevnt underlag.
Grepet på skoene er ganske bra, og de sitter godt i alt bortsett fra dyp gjørme. Har ingen ting å utsette på yttersålen. Aner ikke hvordan den vil holde seg over tid, siden jeg bare har testa skoene i en begrenset tidsperiode.
Overdelen er forsterka på utsatte punkter (på sidene og rundt tærne), så det er bra. Det er også en sånn stretchy bit rundt ankelen, så man slipper å få masse småstein, snø og annet ned i skoa. Dette burde bli vanligere på terrengløpesko!
Forresten så er de ganske store i størrelsen. Jeg bruker 43 1/3 i de aller fleste sko, og testet Flight Vectiv i størrelse 43, og likevel var de egentlig litt for store for meg.
Jeg veide skoa til å være 310 gram per sko. Vanligvis løper jeg i sko som er rundt 200 gram per sko, og jeg synes 310 gram er ganske tungt. Det er ikke sikkert alle bryr seg så mye om dette, eller synes vekta er så høy, men det trekker ihvertfall karakteren opp. Jeg synes uansett det er litt spesielt at det reklameres med at skoa har "lav vekt".
Så, til det viktigste punktet: Karbonfiberplata. Merker ikke noe særlig til denne, må jeg innrømme. Det er uansett vanskelig å måle om den har noen konkret effekt på energibruk og hastighet uten å gjøre skikkelig undersøkelser, men jeg føler ikke at det er noe spesielt å løpe i denne skoa. Sålen er veldig stiv (naturlig nok), og det gjør at jeg den ikke er så behagelig å løpe i oppover bratte motbakker. Produsenten og forhandlere av skoa trekker fram at den har "energigivende egenskaper", og referer til både karbonplata og den såkalte "rockeren", som er den buede formen på sålen. Jeg følte ikke at skoene var noe mer energigivende enn andre løpesko, og hvis den plata utgjør noen forskjell, så er den ihvertfall så minimal at den ikke betyr noe i forhold til andre faktorer som passform, pris, og varighet på materialene.
Helt ærlig så synes jeg ikke skoene var noe særlig, og ihvertfall ikke til en så stiv pris (2199 NOK)! Tipper at skoene uansett selger godt pga karbonfiberplate-hypen. Når det gjelder løpesko så er jo ting utrolig individuelt, og passform og løpsfølelse er (ihvertfall etter min mening) de absolutt viktigste kriteriene. Det kan derfor godt hende at det er mange andre som kommer til å digge disse skoene selv om jeg ikke likte dem! Men er det virkelig verdt å bruke 2199 NOK på disse skoa? Tviler.
Mine erfaringer med skoene ble en del av en artikkel hos Runner's World. De kuttet dessverre en del av kritikken min av skoene, så derfor valgte jeg å publisere denne bloggposten med en komplett tilbakemelding på hva jeg synes om skoa.
Bestemte meg plutselig en dag for å lage meg et hjemmelaga snøbrett. Forutsetningene for dette prosjektet var at det skulle være billig og lett å lage. Når jeg sier "lett å lage", så mener jeg ikke sånn som noen youtubere framstiller det: "Dette er ekstremt lett å lage, du bare går inn i ditt spesialbygde verksted med alle disse spesialverktøyene, og så tar du dette kjempedyre materialet som er umulig å få tak i, og da tar det faktisk bare en ettermiddag før du er ferdig!" Duh...
Hvis det skal være realistisk for meg å gjennomføre, så må jeg bare behøve helt enkle verktøy, som de fleste har liggende i boden. Slik som håndsag, hammer, tommestokk, trelim... Du skjønner greia. Og så kunne jeg selvfølgelig ikke lage et snøbrett med bindinger, for det koster for mye.
Jeg oppdaga at snøbrett uten bindinger er et relativt utbredt konsept, men at det ofte kalles for powder surfer, eller bare pow surfer. Og det er heller ikke så uvanlig å lage dem selv i garasjen. Det er også noen små aktører som produserer powder surfers. Jeg synes spesielt Skorgeboards og The Fyra Board Studio sine brett ser ganske fete ut!
Etter litt søking rundt på internett fant jeg til slutt en video som var midt i blinken for meg: Cheap, Easy, DIY Powder Surfer. Denne videoen gir instruksjoner som til og meg jeg klarer å følge, og krever svært lite av utstyr.
Jeg kjøpte inn en plate med 4 mm kryssfinér av furu. Den er nok ikke like sterk som kryssfinéren han i videoen bruker (baltic birch), men det var det eneste jeg fikk tak i på den lokale byggforhandleren. Når det gjaldt trelim, så turte jeg ikke bruke noe annet enn det som ble anbefalt i videoen, så jeg bestilte en flaske Titebond 3 på nettet.
Jeg sagde finérplaten slik at jeg fikk tre plater på ca 150x40cm. Når trelimet endelig kom i posten, så rigget jeg til noen plankebiter på et gammelt bord, hvor jeg skulle utføre limingen. Plankebitene ble arrangert slik at finérplatene fikk den riktige svaien i front og bak på brettet. Skrutvinger måtte jeg låne, men jeg fikk bare tak i 9 stykker. Skulle nok helst hatt flere for å sørge for at de tre platene ble klemt godt sammen overalt.
Platene ligger klare til å limes, med noen plankebiter under som skal sørge
for at brettet får den riktige svaien.
Jeg smurte på trelim mellom hver av de tre platene før jeg satte på skrutvingene. I ettertid ser jeg at jeg antagelig burde tatt på enda tykkere lag med trelim, men ikke noe å gjøre med det nå...
Her har jeg rigga til platene med skrutvinger mens det trelimet tørker. I
tillegg til skrutvingene la jeg de tyngste tingene jeg fant oppå platene (noen
steiner og hoggstabber). Merk at jeg også hadde lagt de limte platene i mellom
to andre plater for å beskytte selve brettet.
Jeg lot limet tørke i to døgn, før jeg sagde brettet til den formen jeg ville ha. Har veldig liten erfaring med å stå på brett, så her ble det designet litt etter innfallsmetoden. Jeg brukte "Model Tre" fra The Fyra Board Studio som inspirasjon, og brukte ca samme breddemål og lengde som denne. Jeg endte opp med en lengde på 145 cm, og bredde på ca 36/27/28 cm på front/midt/bak.
Slik så det ut etter at limet hadde tørket. Svaien/rockeren ser brukbar ut.
Lagde en papirmal som jeg brukte til å tegne opp på begge sider av brettet,
med utgangspunkt i midtlinja.
Etter å ha pusset kantene godt med sandpapir, så oljet jeg inn hele brettet med en polyuretan-basert gulvlakk. Deretter satte jeg inn undersiden med vanlig glidevoks fra Swix.
Var svært kjekt å ha elektrisk pussemaskin. Men å pusse for hånd er selvsagt
også mulig.
Jeg borret et hull (4mm diameter) helt bak på brettet for å feste en sikkerhetsline i. Denne lina fester jeg i beinet eller beltet så ikke brettet forsvinner når jeg faller.
Feste til sikkerhetsline.
Feste til sikkerhetslina henger fast i brettet ved hjelp av ei knute.
Nesten ferdig, mangler bare noe padding til å stå på.
På oversiden satte jeg på to biter av et skumgummiliggeunderlag, som jeg festet med dobbeltsidig teip.
Slik ser det ferdige brettet ut.
Snøen begynner dessverre i forsvinne her i strøket, så måtte løpe et godt stykke opp på fjellet for å få testa brettet.
Bitene av liggeunderlag som jeg brukte til padding for føttene falt dessverre av etter noen runder, fordi de ikke festet seg godt nok til den dobbeltsidige teipen. Tror jeg skal prøve med en friksjonsteip i stedet (sånn man bruker på trappetrinn etc. for å gi friksjon).
Alt i alt funka brettet bra. Gliden er selvfølgelig ikke like god som vanlige snøbrett, men det er en nok en fordel at det ikke går for fort for min del, så jeg klarer å holde meg på beina...
De første randoneeskia jeg kjøpte var et par brukte Völkl Snowwolf. Jeg hadde lenge stusset på hvorfor så mange tråkka oppover fjellsidene med sånt tungt randoutstyr, når jeg jogga forbi med lærstøvler på lette fjellski, men jeg tenkte at det var på tide å prøve ut sånt utstyr selv. Det ble noen flotte toppturer den våren, men etter at jeg for første gang deltok i en randoneekonkurranse (RandoEnern på Oppdal), skjønte jeg at sko, ski og bindinger av den typen jeg hadde skaffet meg bare var masse dødvekt. Skal det være noe vits i å gå med randoneeski i fjellet, så kan det rett og slett ikke være så tungt og stivt. Skoa hadde så lite gåutslag at jeg av og til ikke merket om de stod å gåmodus eller kjøremodus...
Mitt første sett med randoneeutstyr. Altfor tungt til å være nyttig, men det
var gøy å prøve.
Jeg solgte alt utstyret samme år som jeg kjøpte det, og fikk tak i konkurranseski og -sko på Finn.no. Dynafit PDG-sko fra 2015 (830 gram per sko), og Dynafit Race Performance-ski fra 2011 (990 gram per ski, inkludert binding). Dette er utstyr som det går an å gå på fjelltur med!
Randoneeutstyr laget for konkurransebruk, men det fungerer absolutt til alle
typer turer i vinterfjellet.
Fem sesonger senere så begynner Dynafit-skia å bli passe slitt. Jeg har ingen andre randoneeski å bytte på, så dette utstyret har blitt brukt både til konkurranser og turer. Sålen er rett og slett ganske herpa, og de glir dårlig uansett hvor mye glidsmurning jeg smelter inn. Etter en tur nå i vår oppdaget jeg en skikkelig bulk i sålen, som også gjorde at det kom mye snø innunder fellen, slik at den fort løsna. Det var også brudd i stålkantene flere steder. Jeg tok med skia til en viss skimaker på Oppdal, som avsa en ganske tydelig dødsdom.
Brudd i stålkant på skia, i tillegg til at det begynner å sprekke opp på
oversiden.
Sår i sålen.
Skimakeren oppdaga også noe jeg faktisk ikke hadde lagt merke til selv: Den ene skia var rett og slett knekt i tuppen. Sålen og det øverste laget på skia var fortsatt helt, men kjernen var knekt. Det stod altså mye verre til enn jeg først trodde.
Knekt ski.
Skia skulle få seg et siste farvel: Det ble en tur over Stornubben og Nautgardstind inkludert naboer i Jotunheimen. Den knekte skituppen flagret godt på vei ned fra fjellet, men den holdt stand, til tross for hardt og skavlete føre.
På tur bortover mot Nautgardstind.
Jeg søkte mye rundt i nettbutikker og på Finn.no for å finne en erstatning. Mitt inntrykk er at utvalget av randoneeski til konkurransebruk har blitt litt dårligere enn det var for 4-5 år siden. Kanskje har populariteten for slike konkurranser gått litt ned? Den var vel kanskje ikke så stor i utgangspunktet heller... Jeg fant til slutt et par på tilbud til under 4000 kr: Movement Apple 65. Samme mål som mine gamle ski, men litt lettere. Dessverre tok det litt tid før skia kom i posten, og jeg fikk plutselig muligheten til å ta en tur jeg hadde planlagt en stund: Stå ned en av bekkerennene nordvest på fjellet Tron ved Alvdal. Dermed ble det enda en tur på mine gamle, trofaste Dynafit-ski.
Bekkerenne ned fra Tron.
Skia klarte seg greit ned bekkedalen, men det ble litt ekstra utfordrende på hard skare med svært slitte stålkanter. Jeg stoppet i bunnen av dalen for å ta et blikk opp på svingene jeg hadde kjørt, og da jeg skulle snu meg tilbake oppdaget jeg at skia nå hadde gitt opp ånden helt: Tuppen av ski hadde åpnet seg som en skrellet banan. Fortsatt var det et par kilometer ned til veien, så løsningen ble å teipe sammen tuppen. ALLTID ta med teip på tur.
Det ble definitivt siste tur på dette skiparet.
De nye skia kom i posten dagen etter, og jeg fikk flytta over bindingene samme dag.
Alltid litt nervepirrende å drille hull uten rigg, men det gikk denne
gangen også.
Skia fikk sin jomfrutur i Jotunheimen på vårsnø.
A project I have wanted to for a long time is setting up an "offline Internet", or "Internet-in-a-box", which is an offline "library" and a wireless access point that nearby devices can connect to. To do this I used a Raspberry Pi 4 with the official Raspberry Pi OS installed.
The main steps in creating this "Internet-in-a-box" is:
I assume that the operating system already is installed on the Raspberry Pi (or any similar device).
Scroll down to the bottom of the post to see screenshots from the offline Internet!
Many types of static content (HTML, plain text, pictures etc) can be served from the Raspberry Pi. For me, the most useful content is this:
Other content can be found here.
I use two different ways of serving the content wirelessly to other devices,
one for the .zim-files, and another one for other file types.
For serving .zim-files, I use software called Kiwix. First I download Kiwix
from this link (if you use a
Raspberry Pi like me, choose the version called "GNU/Linux ARM 32-bits").
Unpack the zipped file with
tar -xvzf kiwix-tools_linux-armhf-3.1.2-4.tar.gz
Your version number might be different though. The contents are unpacked to a
folder with the same name as the zipped file. I rename this folder kiwix to
make things easier, and then move all my downloaded .zim-files to this
folder. If you for example download a file called wikipedia.zom and another
file called gutenberg.zim, the folder should look like
kiwix/
├── gutenberg.zim
├── kiwix-manage
├── kiwix-read
├── kiwix-search
├── kiwix-serve
└── wikipedia.zim
First you need to make a library containing your .zim-files:
./kiwix-manage library.xml add gutenberg.zim
./kiwix-manage library.xml add wikipedia.zim
This library can now be served by running:
./kiwix-serve --port=8080 --library library.xml
Go to localhost:8080 in the browser of the Raspberry Pi to see the content.
If the Raspberry Pi is connected to a WiFi-network, you can view the content
from other devices by going to [Local IP of the Pi]:8080 in the browser
(first you need to find the local IP address of the Pi).
For all other file formats I use a Python webserver to serve the content (which of course requires that you have Python installed on your system). This simply means that I run the following command in the folder containing the files I want to serve:
python3 -m http.server 8081
In this case I have chosen to serve on port 8081, but this can be any other port (if you omit the port number, it will be set to 8000) as long it's not the same as the other servers you have set up.
Go to localhost:8081 in the browser of the Raspberry Pi to see the content.
If the Raspberry Pi is connected to a WiFi-network, you can view the content
from other devices by going to [Local IP of the Pi]:8081 in the browser
(first you need to find the local IP address of the Pi).
In order to make the file serving start automatically when I boot the Pi, I simply add cronjobs. Add cronjobs by editing the crontab file:
crontab -e
Add the following files to the bottom:
@reboot /path/to/kiwix/kiwix-serve --port=8080 --library /path/to/kiwix/library.xml
@reboot python3 -m http.server --directory /path/to/files/ 8081
Now these commands will run every time you boot up the Pi.
(The instructions in this section were found in the Raspberry Pi documentation for using the Pi as a routed wireless access point).
After the content is ready to be served from the Pi, one has to make the Pi broadcast its own WiFi. By doing this, any device nearby can connect to the WiFi and access the Pi's content. If you already have a router that all your devices are connected to (including the Pi), you can skip this step.
I used the below commands to make the Raspberry Pi broadcast its own WiFi network.
Install access point software:
sudo apt install hostapd
Enable the accesspoint service and make it start on boot:
sudo systemctl unmask hostapd
sudo systemctl enable hostapd
Provide network management services:
sudo apt install dnsmasq
Utility for firewall rules:
sudo DEBIAN_FRONTEND=noninteractive apt install -y netfilter-persistent iptables-persistent
After installing the software above, I edited the configuration file for
dhcpcd:
sudo vim /etc/dhcpcd.conf
These lines were added to the end:
interface wlan0
static ip_address=192.168.4.1/24
nohook wpa_supplicant
The Pi will be reachable in the IP address 192.168.4.1 after the setup
process is complete, and you are connected to the WiFi of the Pi.
Then I created the hostapd configuration file:
sudo vim /etc/hostapd/hostapd.conf
country_code=<COUNTRY CODE>
interface=wlan0
ssid=<NAME OF NETWORK>
hw_mode=g
channel=7
macaddr_acl=0
auth_algs=1
ignore_broadcast_ssid=0
wpa=2
wpa_passphrase=<PASSWORD>
wpa_key_mgmt=WPA-PSK
wpa_pairwise=TKIP
rsn_pairwise=CCMP
In my case, <COUNTRY CODE> was NO because I live in Norway, and one also have
to fill in a name for the network and password.
After a reboot (sudo systemctl reboot) the Raspberry Pi will broadcast its
own WiFi network, and you can connect to it using the name and password
specified in the hostapd.conf file.
After these tree steps are done, you can connect to the Pi's network using
WiFi, and access the content on 192.168.4.1:8080 or any other port number
that you have chosen to serve your files on. I find this useful both to limit
data usage, but also because I like to work offline if I do not strictly need
to do specific things online.
I'm deeply fascinated with minimalistic and text-only approaches to web design. This January I moved to a place where I have to use a cellphone hotspot to connect to the Internet, and thus I am also very aware of my usage of cellular data, since I need to pay for every GB that I use. This has made me even more conscious of how large many common websites are, and how much useless stuff I'm downloading by just browsing the web.
There seems to be a gaining interest in text-only/low-bandwith websites, as you can read more about in this article. This has several advantages, especially for people that live under conditions with severe limitations on bandwidth. I think it's very cool to see that two major news sites offer low-bandwith versions of their sites:
For most mobile phone companies in Norway (as far as I know), it is common to throttle the bandwidth speed to 64 kbit/s when you have reached your data cap. If more people and companies started to scale down their websites (or at least make low-bandwith versions), one could browse the web comfortably at those kinds of speeds, and websites would be much more accessible. I think this could be very beneficial, especially for essential content such as news.
I just discovered that it is quite easy to make a website adjust the style to whether the client is using dark mode or not. Take a look at a snippet of my CSS for this website:
* {
--txc: #393838;
--lc: #1183b9;
--bgc: #fff;
--hlc: #70b4d5;
}
@media (prefers-color-scheme: dark) {
* {
--txc: #f5f5f5;
--lc: #70b4d5;
--bgc: #212121;
--hlc: #1183b10;
}
}
The first block defines the default values for two text colors, the background and a highlight color. With the second block, those variables are redefined if the user is using a dark system theme.
This might be common knowledge for a lot of people, but have not seen this functionality much "in the wild". Maybe some people prefer to keep the same colors anyway, but I think it is very comfortable that websites dynamically adapt to the system theme, especially when there are lots of articles.
When I first started making my own website, I wrote everything manually, from scratch. This type of approach is very fun and instructional, and probably the best way if one wants to learn how a website functions.
After a while though, this approach gets quite tedious, because the total number of pages piles up (mostly blog posts) and it gets hard to manually update all of them when you want to make changes. This is one of the perfect use cases for a Static Site Generator, which is a tool that uses templates to build the static HTML pages for you. You are able to define the "rules" and layouts for how the page should look independently of the content, and gives you a lot of flexibility with regards to what you want to automate and what you want to do manually.
I resisted the temptation of using an SSG for a while, since I enjoyed writing
everything from scratch and having a minimalistic approach. But, when I wanted
to add an RSS feed to my site, I figured that I couldn't be bothered with
writing an feed.xml file manually. I chose to try out
Hugo, since I didn't really need to install anything, I
could just have a binary executable in the folder where I edit my website, and
then build it there. There was a lot of syntax to learn in order to get
everything working as I wanted, but it was also nice to have RSS feed, tags on
blog posts and blog feed being built automatically.
Even though Hugo worked fine, I felt like it was to bloated and complex for what I wanted. I can certainly recommend Hugo in general, but I want a workflow that has as few dependencies as possible, and is sustainable for a long, long time, with very little maintainance. Keeping it purely HTML+CSS, without a complex build process, would be preferable. I read an article called Why I Built My Own Shitty Static Site Generator, which expressed some of the same thoughts I was having, and I was inspired to stop using Hugo, and restructure this website. I could still use most of what I had written from before, but I wrote my own small script for adding head, header and footer (the elements that are the same on every page) and building the blog feed. After that, the only thing missing was generating an RSS feed. This is probably the most complex part of my build script, but I made it work in the end.
There are some people who are even more hardcore (check out My stack will outlive yours), who avoids having a build step at all. I would like to eliminate the need for a build script, just to keep things even simpler, but I appreciate having the option of easily changing some elements in a template-file rather than on every page. It's also very convinient to write blog posts in Markdown instead of HTML (my build script does this conversion as well). The build script is still quite small and I know exactly what it does and does not, and I prefer that over a more complex SSG for my use case.
After further delving into the world of minimalistic web development, I discovered the 512k.club, the 250kb.club and the 10kclub.com. Those are collections of websites that have a size below a certain (quite small) threshold. I think the goal is just to make a point of how (often needlessly) enormous many websites are, and that it's perfectly possible to make good-looking websites without many MBs of data.
My own main page has a size of 1.6 kB at the time of writing, but of course, there's not much there. Some of the blog posts contain a lot of images, and those pages are certainly bigger in size, but the point is that it's totally fine accessing the website on old browsers and with slow internet speed. I wish that were the case for most websites.
Finally, I see this website as my online "profile page", in the same way that many people have a Facebook-profile as an online presence. Maintaining a website is a little bit harder than making a Facebook-profile, but it gives you much more control and flexibility.
To round off, here are some minimalistic websites that I find interesting (regarding design, content or both):
I have for several months been hosting my own cloud services using Nextcloud on a Raspberry Pi 4. At first I was using a plastic case for the Pi, but I quickly figured out that I had to use something else, as the temperature was constantly in the range of 70-75 degrees Celsius, which I figured was too high when it is supposed to be an always-on system.
I purchased an extra large heat sink and a cluster case with fans in order to reduce the temperature, but I was unsure of how much it would help, and also whether the fan was as quiet as the product description claimed. The delivery of the items took some weeks, so in the meantime I used a stack of coins placed on top of the processing unit, which worked remarkably well (at least compared to my expectations): The temperature was reduced with about 15-20 degrees Celsius.
Using a coin stack as a heat sink.
After attaching the heat sink to the Pi, and then installing the Pi in the cluster case and mounting the fan, the temperature of the Pi dropped to the range of 30-35 degrees Celsius when running a Nextcloud server. This was much better than I expected, and the fans hardly make any noise at all!
Two Raspberry Pis installed in a cluster case, with the old plastic case next
to it.
I have also tried a heatsink case for another one of my Pis, and the temperature is reduced by about 20-25 degrees Celsius, so that is not a bad option either.
I recently got a small keychain multitool called True Utility Fishface, which amongst other things has a bit driver on it. It is basically just a hex-formed hole in the multitool, and can be used to drive any sort of standard screwdriver bit. The Fishface also comes with special dual bit, which is fastened to the multitool with to small rubber bands, and consists of two sizes of the cross-point/Phillips screwdriver head.
The True Utility Fishface.
The Fishface with the dual bit in the driver.
One of the main advantages of carrying a multitool with a bit driver is that it can be adapted for your specific needs. For me it is very useful to carry hex bits, specifically the 4mm and 5mm, since those are commonly used on just about everything on bikes. But how to carry the extra bits I need? One nifty solution would be to drill a hole in the bits, and then they could be fastened directly on the keychain ring, but I do not have the kind of tools needed for that operation. While doing some brainstorming, my eyes landed on a pen on my desk, and suddenly I realized that the shell of that pen was the perfect size for housing a couple of bits.
An identical pen to the one I used for making the bit holder, alongside the
finished "product". The two items are aligned, so you can see where I cut off
the top and bottom of the pen.
After gently sawing the plastic of the pen in the appropriate plazes, using a serrated knife blade, the remaining cylinder could fit the two bits I want to carry around. In one end I fastened a keychain ring with some duct tape, and the other end is closed by screwing the small remaining piece of the bottom part of the pen into place. The bottom part of the bit holder is actually open, but the hole is to small for the bits to fall out.
The exact process of making the bit holder will differ slightly depending on the pen you are using, but it is anyway a quick and cheap way of making a small bit holder with things you most likely already have in your house.
På Strynefjellet og i fjellene over Geiranger er det fortsatt mye snø for tiden, og det var mange som var ute og fikk lufta randoskiene sine opp til Blåtind (1663 moh) i dag. Ikke fullt så mange var det som brukte sykkelen oppover fra Geiranger, men vi var ihvertfall tre stykker som stod klar på fergekaia i morges for å bryne oss på stigningen opp til Dalsnibba. For min del ble det en litt tidligere stopp enn planlagt med punktering etter å ha passert Knuten, men med ekstraslange i sadeltaska kom jeg relativt raskt videre, og mange hårnålssvinger senere ankom jeg Dalsnibba (1476 moh).
Endelig oppe.
Utsikt fra Dalsnibba.
Jeg hadde løpesko med meg, stroppet under sykkelsetet, så turen kunne gå videre i delvis snødekt terreng opp til Fossfjellet, på 1574 moh. Opp dit er det morsom kravling i steinur. Sørover kan man lange ut i slake nedoverbakker, både på snø og isskurt berg, før det går oppover til Rundegga (1615 moh).
Rundegga over Fossvatnet.
Utsikt fra Fossfjellet.
Snøløping på Rundegga.
Fra Rundegga gikk turen tilbake til Dalsnibba, og derfra trillet jeg ned til Djupvasshytta.
Fortsatt fotogene brøytekanter opp Nibbevegen.
På vei ned igjen til Geiranger ble det en liten stopp for å utforske Kvanndalssætra, en samling av gamle seterbygninger som ligger idyllisk til i Kvanndalsbotn, nord for Dalsnibba.
Kvanndalssætra.
Kveldssol over Geiranger.
Det er omtrent 10 år siden jeg første gang hørte om konseptet "Rondane Rundt". Navnet har flere betydninger; for syklister er det først og fremst snakk om ruta rundt Rondane gjennom Grimsdalen og Atnadalen, over Venabygdsfjellet og opp Gudbrandsdalen. For min del er det først og fremst snakk om den beryktede turen over Rondanes ti topper over 2000 moh (med primærfaktor på minst 50 meter), en tur som har blitt mer kjent som "Rondane 2K-maraton".
Opp gjennom åra har mange gjennomført turen i ulikt tempo, og flere har gjennomført på under 12 timer. Den raskeste tida på en slik rute er ikke alltid like lett å fastslå, men det har de siste åra dukket opp mer organiserte prosjekter for å registrere denne type rekorder, som på engelsk kalles fastest known times (FKT). Et slikt prosjekt er fastestknowntime.com, hvor også Rondane Rundt ligger inne under navnet Rondane 2K Marathon. Rekorden har sunket gradvis de siste åra, og så vidt meg bekjent er det tre stykker som har klart å presse tida under 10 timer:
Rutevalg har variert, men det har utviklet seg en slags enighet om at start og mål er Rondvassbu (som også er det mest naturlige stedet å ha som utgangspunkt). I tillegg gjelder det at man skal ha tatt på toppvarden på hver topp, og ikke motta noen støtte underveis. Den tradisjonelle rekkefølgen på toppene er
Fredheim tok en alternativ rute i 2018, og forkortet distansen med noen kilometer ved å løpe ut til Trolltinden først, og deretter tilbake igjen og over Stor- og Veslesmeden, før han fortsatte over Digerronden og de andre toppene i tradisjonell rekkefølge. Da jeg endelig fikk tid og mulighet til å gjennomføre turen selv, valgte jeg å ta samme rute som Fredheim, for å unngå det lange partiet mellom Trolltinden og Digerronden, men jeg ville løpe motsatt vei, slik at jeg kunne skli på snøen ned fra Rondeslottet.
På vei opp fra Rondvassbu.
Jeg syklet inn til Rondvassbu fra Spranget lørdag 27. juni, og kl 8:10 la jeg i vei østover. Opp til Storronden er det tydelig sti hele veien, men jeg prøvde å ta en litt rettere linje noen steder opp langs ryggen. Etter 52 minutter var jeg på toppen, og skulle så ned i skaret mot Vinjeronden. Her hadde jeg aldri vært før, og jeg klarte dessverre å havne for langt øst, slik at jeg måtte løpe litt opp igjen for å komme på rett side av et gjel som går ned i Storbotn.
Stien opp til Storronden.
Mot Vinjeronden.
Vinjeronden og Rondeslottet ble unnagjort raskere enn planlagt, og varmen var heller ikke så slitsom som jeg hadde fryktet. Jeg hadde fått tips på forhånd om at det lå snø omtrent helt ned til Langglupdalen fra Rondeslottet, men det ble litt vinglete rutevalg ned den første biten, da jeg ikke klarte å se hvor snøfonna startet. Etter å løpt noen hundre meter fra toppen, fant jeg en liten bekk hvor jeg fikk fylt ei vannflaske, og like etter på kom jeg endelig inn på snøfonna. Her kunne jeg både skli og løpe nedover, så jeg kom jeg veldig effektivt ned i dalen. Fylte to flasker til i Langglupbekken, før jeg startet på skråningen opp mot Høgronden. Her er det lyng i starten, som kanskje kan oppleves som et kjærkomment avbrekk fra steinura, men jeg synes det var tungt å gå i, da man hele tida synker litt ned for hvert steg.
Utsikt mot Høgronden fra Rondeslottet.
Utsikt fra Høgronden. Her ser man tydelig snøfonna jeg fulgte ned i
dalen.
På toppen av Høgronden kjente jeg vinden for første gang den dagen, og det var deilig å få litt avkjøling. Over til Midtrondene og til Digerronden var beina fortsatt gode, men i steinura ned fra Digerronden begynte det å bli litt tungt. På forhånd hadde jeg planlagt å ta rettest mulig linje ned fra toppen og mot Veslesmeden, men det er såpass mye bratt ur at jeg følte det ble lettere å løpe i en liten bue nedover. Nede i Bergedalen var det deilig å få løpt litt i lyng og på sti, og jeg fylte flaskene flere ganger i bekkene jeg passerte. I tillegg måtte jeg stoppe opp å helle ut stein og grus som hadde samlet seg i skoene på vei ned fra den forrige toppen.
Utsikt fra Høgronden mot Midtrondene.
Etter å ha passert på sørsida av den sydligste Bergedalstjønna, bega jeg meg oppover skråningen som leder opp til ryggen mot Veslesmeden. Farta var ikke like høy som tidligere, men jeg fokuserte på å holde jevn og stabil intensitet. Den siste biten opp mot Veslesmeden innebar litt kravling og lett klyving, og kryssing av et lite bratt snøparti, før jeg kom inn på stien fra Rondvassbu noen timeter fra toppen. Her var det en god del folk, og det hjalp på motivasjonen. Nå gjenstod det bare to topper, men det var også de eneste toppene jeg ikke hadde vært på tidligere, og dessuten de som krever litt mer konsentrasjon for å bestige. Jeg hadde holdt et øye med snittfarta mi, og så vidt jeg kunne se, lå jeg foran skjema til å slå rekorden.
Et lite snøparti rett under Veslesmeden.
Jeg kunne holde grei fart ned i skaret mot Storsmeden, og klyvingen begynte ganske umiddelbart etter at det begynte å gå oppover igjen. Jeg hadde hørt på forhånd at det skulle være noen små varder oppover som viste vei, men jeg var likevel litt usikker på hvor lang tid jeg kom til å bruke opp denne ryggen. Det viste seg å være ganske greit, men jeg brukte litt ekstra tid på å vurdere hvert veivalg for å være sikker på at jeg ikke kastet bort tid på klyve opp feil sted. Akkurat i det jeg kom opp det siste klyvepartiet, kjente jeg de første dråpene. Regnjakka kom raskt på, men det gikk ikke raskt ned fra Storsmeden; skoa mistet helt grepet på steinene. Flere ganger skled jeg i ura, og en av gangene slo jeg siden av hodet i en stein. Etter en liten pause konkluderte jeg med at det hadde gått fint, men det ble en påminnelse på hvor lite som skal til før det kan gå ordentlig galt!
Fra Veslesmeden mot Storsmeden.
Ut mot toppen på 1897 prøvde jeg å holde meg litt nede i fjellsida for å slippe å gå opp unødvendige høydemeter. Her bestod terrenget av små, skråstilte sva, som var svært glatte å forsere når det hadde blitt bløtt. Jeg angra på rutevalget, og tenkte at på vei tilbake skulle jeg gå lengre opp, hvor jeg kunne holde høyere fart. Ryggen utover var ganske morsom, men jeg var spent på hvordan den siste biten kom til å være, da jeg hadde lest noen sprikende turrapporter på nettet i forkant. Det er sikkert mulig å ta flere forskjellige opp langs ryggen, men den mest brukte stien var veldig tydelig, og også merket med små varder. Jeg møtte et par stykker i det siste skaret på ryggen, som tipset meg om at det kunne være litt vanskelig å finne veien ned igjen, da det er flere grusrenner som kan se ut som stier. Dette var nyttig informasjon, og jeg fulgte ekstra godt med hvor vardene befant seg, slik at jeg effektivt kunne løpe ned igjen uten å måtte stoppe og vurdere veivalg.
Oppe på Trolltinden kunne jeg meget fornøyd konstatere at dette var dagens siste topp, men jeg kjente på at det fortsatt var en del kilometer igjen. Jeg så på snittfarta mi, og begynte å ane at rekorden hang i en meget tynn tråd. Tilbake over ryggen var det vanskelig å presse farta opp noe særlig, da jeg først og fremst måtte fokusere på å komme meg trygt fram på slitne bein. Men da jeg skulle ned i skaret over Kaldbekkbotn kom jeg inn på en snøfonn, og håpet om å ta rekorden kom tilbake. Fra Kaldbekkbotn og til Rondvassbu er det sti hele veien, og jeg fikk endelig strekke ut beina litt. Minuttene gikk kjapt, og jeg skjønte etterhvert at dette kom til å bli utrolig jevnt. Motivasjonen var høy de siste par kilometerne, men samtidig var beina veldig tunge. I det jeg kom inn på grusveien mot Rondvassbu, var tida 8:47, men GPS-klokka viste bare timer og minutter, så jeg hadde ikke oversikt over hvor lang tid jeg hadde på meg. Jeg spurta det jeg klarte, og lettelsen var stor da jeg stoppa klokka idet jeg ankom hovedinngangen på Rondvassbu, og så at tida var på 8:47:52.
Fremme ved Rondvassbu etter 8 timer og nesten 48 minutter.
Utafor hytta satt det noen vennlige sjeler som hentet et stort glass vann til meg mens jeg fikk igjen pusten. Jeg fikk ikke med meg navnene deres, men tusen takk for hjelpen! Etter å ha summet meg litt, gikk turen tilbake til Spranget på sykkel.
Digg å være ferdig.
Under ligger et kart over ruta jeg tok. Oversikt over både dame- og herrerekord finnes her, og link til økta finnes her.
Kart over ruta.
Some interesting revelations on Facebook's targeted advertising.
This isn’t the first time Facebook has faced scrutiny for the ways its ad tool can target conspiracy-minded people. A 2019 Guardian investigation found that advertisers could reach people interested in “vaccine controversies.” In 2017, ProPublica reporters (two of whom are now at The Markup) found that advertisers could target people who were interested in terms like “Jew hater,” and “History of ‘why jews ruin the world.’ ”
Read Want To Find A Misinformed Public? Facebook's Already Done It.
Tidlig i mars hadde det kommet unormalt mye snø i Rondane, og jeg utnytta muligheten til å dra på topptur i området. Planen var egentlig å få gått en runde over noen av toppene over 2000 moh, men mye dyp snø og skredfare gjorde at jeg snudde ved Midtronden, og gikk samme vei tilbake.
Nederst på sida ligger en video fra turen.
På vei oppover bjørkeskogen.
På toppen av Høgronden med utsikt mot Rondeslottet.
De to Midtrondene i sikte.
Flanken til høyre hadde jeg opprinnelig tenkt å stå ned, men der var det
På toppen av Midtronden Aust, med utsikt mot Midtronden Vest.
Sola nærmer seg horisonten, rett over Rondeslottet.
Nede igjen ved Gammelgarden.
View from Na Gruagaichean, the finish line of Mamores VK.
The final ridge of Mamores VK.
Scottish sunset 1.
Scottish sunset 2.
Scottish sunset 3.
The gear I used during Glen Coe Skyline (I brought only some of the food
shown in the picture.)
Nervous atmosphere before the start of Glen Coe Skyline.
I was running at the tail of the lead pack during the first kilometers
of the race.
This was the last picture I took during the race. At this point I
understood that things were going much better than expected, and I
needed to stop fumbling around with my GoPro camera.
Finally done.
Can\'t describe the feelings I had at this moment.
A trophy and money. Best prize ever.
Rather happy to be on that list of names. Photo credit: Skyline
Scotland.
This is not open sourcing any actual software or hardware it is “open-sourcing the API documentation for its SoundTouch smart speakers”. You might be able to point them at an alternative back-end¹ if you want the cloud features, but that will need to be written from scratch rather than being forked from code provided by Sonos.
> When cloud support ends, an update to the SoundTouch app will add local controls to retain as much functionality as possible without cloud services
This is a far bigger move than releasing API information, IMO bigger than if they had actually open sourced the software & hardware, from the point of view of most end users - they can keep using the local features without needing anyone else to maintain a version.
--------
[1] TFA doesn't state that this will be possible, but opening the API makes no sense if it isn't.
This def needs to be celebrated and rewarded. I am more likely to purchase Bose now.
This is how "end of support" should be handled. Instead of turning devices into e-waste, open-source them and let the community extend their life. Kudos to Bose for setting a good example.
More companies should follow this approach - especially as right-to-repair becomes a bigger issue.
Hey! I created Jeff Dean Facts! Not the jokes themselves, but the site that collected them.
It was in 2008 I think (give or take a year, can't remember). I worked at Google at the time. Chunk Norris Facts was a popular Internet meme (which I think later faded when he came out as MAGA, but I digress...). A colleague (who wishes to remain anonymous) thought the idea of Jeff Dean Facts would be funny, and April 1st was coming up.
At the time, there was a team working on an experimental web app hosting platform code named Prometheus -- it was later released as App Engine. Using an early, internal build I put together a web site where people could submit "facts" about Jeff Dean, rate each other's facts on a five-star scale, and see the top-rated facts. Everything was anonymous. I had a few coworkers who are funnier than me populate some initial facts.
I found a few bugs in Prometheus in the process, which the team rapidly fixed to meet my "launch date" of April 1st. :)
On the day, which I think was a Sunday, early in the morning, I sent an email to the company-wide "misc" mailing list (or maybe it was eng-misc?) from a fake email address (a google group alias with private membership), and got the mailing list moderator to approve it.
It only took Jeff an hour or two to hack his way through the back-end servers (using various internal-facing status pages, Borg logs, etc.) to figure out my identity.
But everyone enjoyed it!
My only regret is that I targeted the site specifically at Jeff and not Sanjay Ghemawat. Back then, Jeff & Sanjay did everything together, and were responsible for inventing a huge number of core technologies at Google (I have no idea to what extent they still work together today). The site was a joke, but I think it had the side effect of elevating Jeff above Sanjay, which is not what I intended. Really the only reason I targeted Jeff is because he's a bit easier to make fun of personality-wise, and because "Jeff Dean Facts" sort of rolls off the tongue easier that "Sanjay Ghemawat Facts" -- but in retrospect this feels a little racist. :(
My personal favorite joke is: Jeff Dean puts his pants on one leg at a time, but if he had more than two legs, you'd see his approach is actually O(log n).
This thread reads like an advertisement for ChatGPT Health.
I came to share a blog post I just posted titled: "ChatGPT Health is a Marketplace, Guess Who is the Product?"
OpenAI is building ChatGPT Health as a healthcare marketplace where providers and insurers can reach users with detailed health profiles, powered by a partner whose primary clients are insurance companies. Despite the privacy reassurances, your health data sits outside HIPAA protection, in the hands of a company facing massive financial pressure to monetize everything it can.
https://consciousdigital.org/chatgpt-health-is-a-marketplace...
> grass fed, free range... Because agribusiness doesn't make money with those.
Agribusiness absolutely makes money off of those. In fact they had a hilariously easy time adapting to the consumer trend because all they had to do to label a cow “free range” or “grass fed” was change the finishing stage to a lower density configuration instead of those abominable feed lots you see along highways. The first two stages, rearing and pasturing, didn’t change because they were already “free range” and “grass fed”. Half of the farmland in the US is pastureland and leaving animals in the field to eat grass was always the cheapest way to rear and grow them. They only really get fed corn and other food at the end to fatten them up for human consumption.
The dirty not-so-secret is that free range/grass fed cows eat almost the exact same diet as regular cows, they just eat a little more grass because they’re in the field more during finishing. They’re still walking up to troughs of feed, because otherwise the beef would be unpalatable and grow quite slower.
True grass fed beef is generally called “grass finished” beef and it’s unregulated so you won’t find it at a supermarket. They taste gamier and usually have a metallic tang that I quite honestly doubt would ever be very popular. The marbling is also noticeably different and less consistent. Grain finished beef became popular in the 1800s and consumers in the West have strongly preferred it since.
I’m not sure you can even find a cow in the entire world that isn’t “grass fed”. Calves need the grass for their gut microbiomes to develop properly.
Before the "rewrite it in Rust" comments take over the thread:
It is worth noting that the class of bugs described here (logic errors in highly concurrent state machines, incorrect hardware assumptions) wouldn't necessarily be caught by the borrow checker. Rust is fantastic for memory safety, but it will not stop you from misunderstanding the spec of a network card or writing a race condition in unsafe logic that interacts with DMA.
That said, if we eliminated the 70% of bugs that are memory safety issues, the SNR ratio for finding these deep logic bugs would improve dramatically. We spend so much time tracing segfaults that we miss the subtle corruption bugs.
Tyson foods and other meatpacking companies lobbied and funded RFK...
Here's industry reports
https://www.nationalbeefwire.com/doctors-group-applauds-comm...
https://www.wattagnet.com/business-markets/policy-legislatio...
And straight up lobbying groups
https://www.nationalchickencouncil.org/new-dietary-guideline...
https://www.meatinstitute.org/press/recommend-prioritizing-p...
Lobbying groups, putting out press releases, claiming victory...
Here's some things you won't find in any of the documents, including the PDFs at the bottom: community gardens, local food, farmers markets, grass fed, free range... Because agribusiness doesn't make money with those.
Just because you might like the results doesn't mean they aren't corrupt as hell
My uncle had an issue with his balance and slurred speech. Doctors claimed dementia and sent him home. It kept becoming worse and worse. Then one day I entered the symptoms in ChatGPT (or was it Gemini?) and asked it for the top 3 hypotheses. The first one was related to dementia. The second was something else (I forget the long name). I took all 3 to his primary care doc who had kept ignoring the problem, and asked her to try the other 2 hypotheses. She hesitantly agreed to explore the second one, and referred him to a specialist in that area. And guess what? It was the second one! They did some surgery and now he's fine as a fiddle.
It's not that simple - the problem is that those institutions are market makers. They are a tiny portion of the market, but a huge driving force in setting and manipulating prices, because their properties get leveraged, instrumentalized, and securitized, with derivative products, speculation, and all sorts of incentives that you don't normally want operating in the arena of housing.
The things that they do have massively outsized downstream impact contrasted against their relatively tiny overall participation in the market, and they can afford to behave in ways that manipulate the behavior of the majority.
If you can decouple them from the housing markets, you also decouple the interests of the donor class, and you allow for policy that doesn't maximize the cost of real estate over the interests of the majority of the population.
> the US's per capita consumption of meat
That number seemed unreal to me, so I looked it up. I think it represents the total pre-processing weight, not the actual meat meat consumption. From Wikipedia:
> As an example of the difference, for 2002, when the FAO figure for US per capita meat consumption was 124.48 kg (274 lb 7 oz), the USDA estimate of US per capita loss-adjusted meat consumption was 62.6 kg (138 lb)
Processing, cutting into sellable pieces, drying, and spoilage/loss mean the amount of meat consumed is about half of that number.
I'm one of the Tailscale engineers who built node state encryption initially (@awly on Github), and who made the call to turn it off by default in 1.92.5.
Another comment in this thread guessed right - this feature is too support intensive. Our original thinking was that a TPM being reset or replaced is always sign of tampering and should result in the client refusing to start or connect. But turns out there are many situations where TPMs are not reliable for non-malicious reasons. Some examples: * https://github.com/tailscale/tailscale/issues/17654 * https://github.com/tailscale/tailscale/issues/18288 * https://github.com/tailscale/tailscale/issues/18302 * plus a number of support tickets
TPMs are a great tool for organizations that have good control of their devices. But the very heterogeneous fleet of devices that Tailscale users have is very difficult to support out of the box. So for now we leave it to security-conscious users and admins to enable, while avoiding unexpected breakage for the broader user base.
We should've provided more of this context in the changelog, apologies!
Of note: the US's per capita consumption of meat has increased by more than 100 pounds over the last century[1]. We now consume an immense amount of meat per person in this country. That increase is disproportionately in poultry, but we also consume more beef[2].
A demand for the average American to eat more meat would have to explain, as a baseline, why our already positive trend in meat consumption isn't yielding positive outcomes. There are potential explanations (you could argue increased processing offsets the purported benefits, for example), but those are left unstated by the website.
[1]: https://www.agweb.com/opinion/drivers-u-s-capita-meat-consum...
[2]: https://ers.usda.gov/data-products/chart-gallery/chart-detai...
It will make very little difference in the end.
Australia's land tax system makes it effectively impossible for large corporations to own large chunks of residential property, but our real estate is amongst the world's most expensive and landlords are still awful - it's just that the landlords are hundreds of thousands of dentists and, yes, software engineers rather than corporate entities.
If you want housing to be cheaper and renters to be better treated, increase supply. Everything else is window-dressing.
For all the lunacy of RFK this somehow is actually a really good set of guidelines? Certainly better than the previous version. I didn't expect that to be honest.
The key word here is "Wall Street". And this statement is playing off a popular misconception around corporate investors buying up American houses.
There has been a bit of a panic around "Investors buying up all the property!!!" With people often citing Black Rock and Blackstone as the main culprits. But most of the "investors" buying up property are individuals purchasing investment properties.
Here's an article on the topic from 2023[0], a bit old but my understanding is large institutional investment in residential real estate was already starting to cool down.
Black rock isn't buying up all the housing, your neighbors are.
I suspect this statement, and even if it becomes an actual ban, is largely to gain wider popular support around a largely imaginary concern people have.
0. https://www.housingwire.com/articles/no-wall-street-investor...
Speaking from personal experience, this is consistent with multiple doctors over the years recommending high-protein, low carb diets. (Clarification: low does not mean no carb.)
I don't understand people freaking out over this - outside of a purely political reflex - hell hath no fury like taking away nerds' Mountain Dew and Flamin' Hot Cheetos.
Nor do I understand the negative reactions to new restrictions on SNAP - candy and sugary drinks are no longer eligible.
Tools like Tailwind are one of the few cases where I totally believe it when the CEO says "we are cutting jobs because of AI".
Sucks that anytime you ask AI to generate a site for you Tailwind will have an impact on that.
> It's not that Dell doesn't care about AI or AI PCs anymore, it's just that over the past year or so it's come to realise that the consumer doesn't.
I wish every consumer product leader would figure this out.
The paid products Adam mentions are the pre-made components and templates, right? It seems like the bigger issue isn't reduced traffic but just that AI largely eliminates the need for such things.
While I understand that this has been difficult for him and his company... hasn't it been obvious that this would be a major issue for years?
I do worry about what this means for the future of open source software. We've long relied on value adds in the form of managed hosting, high-quality collections, and educational content. I think the unfortunate truth is that LLMs are making all of that far less valuable. I think the even more unfortunate truth is that value adds were never a good solution to begin with. The reality is that we need everyone to agree that open source software is valuable and worth supporting monetarily without any value beyond the continued maintenance of the code.
> But the reality is that 75% of the people on our engineering team lost their jobs here yesterday because of the brutal impact AI has had on our business.
Adam is simply trying to navigate this new reality, and he's being honest, so there's no need to criticize him.
Very sad to hear, I bought Tailwind UI years ago and although it was a lot more expensive than I wanted, I've appreciated the care and precision and highly recommend buying it (It's now called Tailwind Plus) even still (maybe even especially now).
Mad props to Adam for his honesty and transparency. Adam if you're reading, just know that the voices criticizing you are not the only voices out there. Thanks for all you've done to improve web development and I sincerely hope you can figure out a way to navigate the AI world, and all the best wishes.
Btw the Tailwind newsletter/email that goes out is genuinely useful as well, so I recommend signing up for that if you use Tailwind CSS at all.
You’re exactly right: This one incident did not shape the entire body of scientific research.
There is a common trick used in contrarian argumentation where a single flaw is used to “debunk” an entire side of the debate. The next step, often implied rather than explicit, is to push the reader into assuming that the opposite position must therefore be the correct one. They don’t want you to apply the same level of rigor and introspection to the opposite side, though.
In the sugar versus saturated fat debate, this incident is used as the lure to get people to blame sugar as the root cause. There is a push to make saturated fat viewed as not only neutral, but healthy and good for you. Yet if you apply the same standards of rigor and inspection of the evidence, excess sugar and excess saturated fat are both not good for you.
There is another fallacy in play where people pushing these debates want you to think that there is only one single cause of CVD or health issues: Either sugar, carbs, fat, or something else. The game they play is to point the finger at one thing and imply that it gets the other thing off the hook. Don’t fall for this game.
It makes sense that you wouldn't hire in such an uncertain environment. We have a President using emergency powers to affect sweeping, unpredictable, consequential changes to the economy that can dramatically alter unit economics overnight and completely tank a previously viable business. Within this calendar year, the President's ability to do this may be upended by pending court cases, an election, or both. Following those potential changes, the breach of trust created by the previous chaos may mean that trade never returns to normal. I don't envy anyone trying to make long-term business decisions, like hiring, in such an environment.
When this news first came out it was mind blowing, but at the same time I don't entirely get it.
So the money quote seems to be:
> The literature review heavily criticized studies linking sucrose to heart disease, while ignoring limitations of studies investigating dietary fats.
They paid a total of 2 people $50,000 (edit: in 2016 dollars).
That doesn't seem like enough to entirely shape worldwide discourse around nutrition and sugar. And the research was out there! Does everybody only read this single Harvard literature review? Does nobody read journals, or other meta studies, or anything? Did the researchers from other institutions whose research was criticized not make any fuss?
I guess the thing that I most don't get is it's now been 10 years since then, and I haven't seen any news about the link between sugar and CVD.
> There is now a considerable body of evidence linking added sugars to hypertension and cardiovascular disease
Okay, where is it? What are the conclusions? Is sugar actually contributing more than fat for CVD in most patients? Edit: Or, is the truth that fat really is the most significant, and sugar plays some role but it's strictly less?
Nice! The author touches on the area properties and here's the most practical life hack derived from the standard I personally use. It uses the relationship between size and mass.
Because A0 is defined as having an area of exactly 1 square meter, the paper density (GSM or grams per square meter) maps directly to the weight of the sheet.
>A0 = 1 meter square.
>Standard office paper = 80 gsm
>Therefore, one sheet of A0 = 80 grams.
>Since A4 is 1/16th of an A0, a single sheet of standard A4 paper weighs 5 grams.
I rarely need to use a scale for postage. If I have a standard envelope (~5g) and 3 sheets of paper (15g), I know I'm at 20g total. It turns physical shipping logistics into simple integer arithmetic. The elegance of the metric system is that it makes the properties of materials discoverable through their definitions.
I'm tired of constantly debating the same thing again and again. Where are the products? Where is some great performing software all LLM/agent crafted? All I see is software bloatness and decline. Where is Discord that uses just a bunch of hundreds megs of ram? Where is unbloated faster Slack? Where is the Excel killer? Fast mobile apps? Browsers and the web platform improved? Why Cursor team don't use Cursor to get rid of vscode base and code its super duper code editor? I see tons of talking and almost zero products.
I too would very happily do just the bits of my job that I like, when and how I want, and have any requests or comments or complaints I make get immediate attention and responses.
All in the knowledge that no one is going to be time-tracking me or doing performance reviews, and I can just not do work at any moment I don't feel like it or have something better to do that day, like go to my private island or take my private jet to burning man etc (or as it turns out do a talk at Stanford). All while you have so much money that the price of anything from clothes to cars to houses is just some arbitrary number that has no meaning to you it is so absolutely tiny number... not that you actually buy anything yourself any more, mainly your team of personal staff deal with that grubby reality.
As for the rest of us, well we need to pay the bills while playing "the game" and politics and cowtowing to keep the money coming.
This is interesting to hear, but I don't understand how this workflow actually works.
I don't need 10 parallel agents making 50-100 PRs a week, I need 1 agent that successfully solves the most important problem.
I don't understand how you can generate requirements quicky enough to have 10 parallel agents chewing away at meaningful work. I don't understand how you can have any meaningful supervising role over 10 things at once given the limits of human working memory.
It's like someone is claiming they unlocked ultimate productivity by washing dishes, in parallel with doing laundry, and cleaning their house.
Likely I am missing something. This is just my gut reaction as someone who has definitely not mastered using agents. Would love to hear from anyone that has a similar workflow where there is high parallelism.
What bothers me about posts like this is: mid-level engineers are not tasked with atomic, greenfield projects. If all an engineer did all day was build apps from scratch, with no expectation that others may come along and extend, build on top of, or depend on, then sure, Opus 4.5 could replace them. The hard thing about engineering is not "building a thing that works", its building it the right way, in an easily understood way, in a way that's easily extensible.
No doubt I could give Opus 4.5 "build be a XYZ app" and it will do well. But day to day, when I ask it "build me this feature" it uses strange abstractions, and often requires several attempts on my part to do it in the way I consider "right". Any non-technical person might read that and go "if it works it works" but any reasonable engineer will know that thats not enough.
How are you, hacker?
🪐 What’s happening in tech today, January 8, 2026?
The HackerNoon Newsletter brings the HackerNoon homepage straight to your inbox. On this day, The No Child Left Behind Act was signed into law in 2002, President Washington delivered the first State of the Union Address in New York City in 1790, Elvis Presley was born in 1935, and we present you with these top quality stories. From Gran Turismo 2026: The Great AI Showdown for Autonomous Driving to 5 Open-Source Fashion Projects to Use for Free —and Support with Kivach, let’s dive right in.
By @vinitabansal [ 12 Min read ] Here are the five critical dysfunctions every leadership team must guard against to unlock unwavering trust Read More.
By @dmtrmrv [ 10 Min read ] Start with markup, not styles. Write only the CSS you actually need. Design for mobile first, not as a fix later. Let layouts adapt before reaching for breakpoi Read More.
By @obyte [ 6 Min read ] Open-source meets style: let’s explore some free fashion tools and learn how to support them with crypto donations through Kivach. Read More.
By @zbruceli [ 20 Min read ] Tesla has the data ocean. Waymo has the ground truth. Nvidia has the compute. Who wins the Gran Truismo of autonomous driving? Read More.
By @hackernoon-courses [ 3 Min read ] Learn how to write in your own voice, build confidence in your style, and create work that sounds unmistakably you. Read More.
🧑💻 What happened in your world this week?
It's been said that writing can help consolidate technical knowledge, establish credibility, and contribute to emerging community standards. Feeling stuck? We got you covered ⬇️⬇️⬇️
ANSWER THESE GREATEST INTERVIEW QUESTIONS OF ALL TIME
We hope you enjoy this worth of free reading material. Feel free to forward this email to a nerdy friend who'll love you for it.See you on Planet Internet! With love, The HackerNoon Team ✌️
Nvim 0.11 was released.
Neovim is participating in Google Summer of Code 2025!
If you’re a student and want to work on Neovim for GSoC, start here.
The next steps are:
Thank you for your interest in Neovim!
Nvim 0.10 was released.
Bram is one of my heroes. That’s literal and recursive: when I say it, internally I check before making a frivolous claim, which is a feature of this particular role-model; “What would Bram do?” is a fixture in me which informs my choices.
Those who studied vim_dev and the Vim source and docs, accumulated treasure
from a stream of copious messages and spare impressions. But also from what he
omitted: he never appealed to sensationalism or personal judgements.
Neovim is the world’s most-loved editor. That’s just science:
Here are some highlights from Neovim 2022 (Nvim 0.8) development.
Eye candy first!
:set laststatus=3
'winbar' is like an extra statusline at the top of each window. It complements laststatus=3:
set winbar=%f
set laststatus=3
'winbar' and 'statusline' gained support for mouse-click regions (as ’tabline’ has had since 2016):
:set cmdheight=0
:set mousescroll=ver:5,hor:2
:set rnu nu
:let &stc='%#NonText#%{&nu?v:lnum:""}%=%{&rnu&&(v:lnum%2)?"\ ".v:relnum:""}%#LineNr#%{&rnu&&!(v:lnum%2)?"\ ".v:relnum:""}'
:set jumpoptions=view
vim.lsp.start({ name = 'godot', cmd = vim.lsp.rpc.connect('127.0.0.1', 6008) })
LspAttach, LspDetach. Example:
vim.api.nvim_create_autocmd('LspAttach', {
group = yourGroupID,
callback = function(args)
local client = vim.lsp.get_client_by_id(args.data.client_id)
your_callbac_func(client, args.buf)
end
}
vim.lsp.get_active_clients() learned to filter (this will be a standard pattern in the Lua stdlib):
get_active_clients({id=42})
get_active_clients({bufnr=99})
get_active_clients({name='tsserver'})
:set diffopt+=linematch:60
vim.g.editorconfig_enable = false
'inccommand' feature (since 2017), which show the effects of :substitute (:s/foo/bar) as you type.:normal and macros:
:normal, :global, etc.
vim.api.nvim_create_user_command(
'MyCmd',
my_cmd,
{ …, preview = my_cmd_preview })
:write command gained the ++p flag, so this creates parent/dir/ if it doesn’t exist:
:edit parent/dir/file.txt
:write ++p
$XDG_STATE_HOME ($XDG_CACHE_HOME (stdpath('log') to get the recommended location for log files.:help :Man) shows an outline (table of contents) in the location list. Now the outline also lists the flags.
before:
9.0ms: sourcing …/runtime/filetype.vim
after:
1.3ms: sourcing …/runtime/filetype.lua
nvim --startuptime now reports Lua require() times.
000.010 000.010: --- NVIM STARTING ---
000.198 000.188: event init
...
026.333 001.109 001.101: require('vim.lsp.protocol')
028.144 000.423 000.423: require('vim.lsp._snippet')
...
mouse=nvi
Type ":" (cmdline-mode) to temporarily disable mouse. Right-click shows a popup menu.
Try it!
nvim_cmd([list]) and “user cmd-preview”! And super useful for defining custom cmdline (:) behavior.
:echo nvim_parse_cmd('.,$g/foo/bar', {})
{
'cmd': 'global',
'args': ['/foo/bar'],
'mods': {…},
'magic': {'file': v:false, 'bar': v:false}
}
nvim_cmd() to call any Vim legacy command in a structured way, like system([...]).
fnameescape(): special chars are controlled by the magic param.
nvim_cmd({cmd='vimgrep', args={'/%s/j', '**'}}, {})
vim.fs.find() is now the canonical way to find “root files”, common for LSP configuration.vim.cmd is the Lua nvim_cmd wrapper. It supports calling Ex commands as functions instead of strings:
vim.cmd.colorscheme('nightfox')
“Lua plugins are basically the same as a vim plugin, except the file extension is
.luainstead of.vimand the file contains Lua code instead of Vimscript.”
Original article: https://gpanders.com/blog/whats-new-in-neovim-0-7
Neovim 0.7 was just released, bringing with it lots of new features (and of course plenty of bug fixes). You can find the full release notes here, but in this post I’ll cover just a few of the new additions.
Neovim 0.5 saw the introduction of Lua as a first-class citizen in the Neovim
ecosystem: Lua could now be used in the user’s init file, plugins,
colorschemes, ftplugins, etc. Basically, anywhere that you could use a .vim
file, you could now use .lua instead.
The real 0.5 was the friends we made along the way
The long-awaited release of Neovim v0.5.0 finally happened on July 2, 2021. It was worth the wait: With over 4000 commits, it is so big that it broke some of the release tooling.
These notes focus on the most user-visible improvements, of which the biggest are:
The LWN article also is a good overview.
Neovim is a fork of the venerable text-editor vim, focused on extensibility and usability. It is not a rewrite but a continuation and extension of Vim. Many clones and derivatives exist, some very clever—but none are Vim. Neovim is built for users who want the good parts of Vim, and more. See a list of differences via :help vim-differences.
If you are interested in contributing to Neovim, visit the github page for details. We have a great test suite and automated CI, so you can just jump in and have a go. Don’t be afraid to open a PR to get comments or just take advantage of the infrastructure!
Neovim is a fork of the venerable text-editor vim, focused on extensibility and usability. It is not a rewrite but a continuation and extension of Vim. Many clones and derivatives exist, some very clever—but none are Vim. Neovim is built for users who want the good parts of Vim, and more. See a list of differences via :help vim-differences.
If you are interested in contributing to Neovim, visit the github page for details. We have a great test suite and automated CI, so you can just jump in and have a go. Don’t be afraid to open a PR to get comments or just take advantage of the infrastructure!
Neovim was again accepted into the Google Summer of Code program. View our page on the GSoC website for details. Our ideas list runneth over! You’re also welcome to propose other ideas in chat or on the ticket tracker.
In our first GSoC ever, we mentored two students who both completed their projects.
Brock Mammen implemented a C# client for Nvim, and even went further with a Nvim extension for Visual Studio! He also made significant improvements to the build system, particularly for Windows/MSVC.
Neovim is a fork of the venerable text-editor vim, focused on extensibility and usability. It is not a rewrite but a continuation and extension of Vim. Many clones and derivatives exist, some very clever—but none are Vim. Neovim is built for users who want the good parts of Vim, and more. See a list of differences at :help vim-differences.
If you are interested in contributing to Neovim, read CONTRIBUTING.md to get started. We have a great test suite and automated CI, so you can just jump in and have a go. Don’t be afraid to open a PR to get comments or just take advantage of the infrastructure!
It’s time for the Neovim newsletter! Skip to the Fun and Features sections if you only care about new features. There are also changes in project management that you should know about.
Each minute, a new text editor is born (source: Hacker News). There are endless text editors that address the “common case”.
Instead of another Vim-like text editor, Neovim users want a better Vim. Thousands of small decisions live in the Vim core, accumulated over decades; most of those decisions are still relevant, solving subtle problems that new projects have yet to encounter, digest and decide.
Welcome to the sixth newsletter for Neovim, a project that aims to improve Vim by adding new features and wrap it all in a nice, modern face.
Hi, this is @tarruda and I will be addressing the Neovim community directly in this newsletter. Other than that, I will try to keep it structured as @jdavis did previously. Let’s get started!
Neovim now has its first public release!
Outstanding communication and handling of bugs from Neovim. Exemplary FOSS project in my experience so far. @telemachus
Welcome to the fifth newsletter for Neovim, the project that looks to become Vim out of the box.
Now in addition to the mailing list and IRC channel, you can hop into the Neovim chat room provided through Gitter.im. It has a convenient web interface but also allows you to connect using your favorite IRC client.
The way to “win” at open source is to be consistent and steady.
Welcome to the fourth newsletter for Neovim, the project that aims to overhaul the Vim code base. It also hopes to provide out of the box features yet a more powerful interface for extending the editor.
You can now add a Neovim sticker to your laptop by heading over to the Neovim section of Unixstickers.
Welcome to the third (and delayed) newsletter for Neovim, the project that aims to overhaul the Vim codebase.
The next newsletter will be coming out near the beginning of November. It has changed to be every other month as to not overload any sensitive Neovim lovers with too much staggering awe.
The Issue tracker on GitHub now has over 1,000 issues/pull requests. The lucky 1000th pull request was opened shortly before August.
Welcome to the second monthly newsletter for Neovim, the project that dreams to be the next text editor that we’ll all love.
In the discussion about default Neovim settings, the idea of having a default colorscheme was brought up. A later issue was started to determine what colorscheme would be used.
It was decided that a new colorscheme would be great for Neovim. To do this, the Neovim collaborators will create a contest that will allow submissions of colorschemes. After some time, the submissions will close and then a new Poll will be started to let the community pick the winner.
Welcome to the first newsletter for Neovim, a project that hopes to give a new beginning to a text editor that we all love.
We asked and the support was overwhelming; the community wanted a newsletter.
The plan is to release a newsletter each month to detail the progress and anything else newsworthy for the project.
Future newsletters will be released on the first Friday of every month. That makes the next one scheduled for July 4th.
git clone https://github.com/neovim/neovimcd neovim
git checkout stable.make CMAKE_BUILD_TYPE=RelWithDebInfo
CMAKE_INSTALL_PREFIX. See also ../install.gmake instead of make.sudo make install
/usr/localsudo make install, you can try cd build && cpack -G DEB && sudo dpkg -i nvim-linux-<arch>.deb (with <arch> either x86_64 or arm64) to build DEB-package and install it. This helps ensure clean removal of installed files. Note: This is an unsupported, “best-effort” feature of the Nvim build.Notes:
You can install Neovim from download, package, or source in just a few seconds.
nvim (not neovim).
Downloads are available on the Releases page.
Packages are listed below. (You can also build Neovim from source.)
Windows 8+ is required. Windows 7 or older is not supported.
The roadmap is an overview of the project direction. Detailed plans and priorities are tracked in milestones (these are tentative and may be changed or dropped at any time):
0.1, 0.2, …) track production releases. The next upcoming version has a (estimated) target date.Concrete high-level feature areas and changes.
pkg.jsonnvim_on()vim.async--remotevim.lsp.server()vim.pack) #34009:connect, :restart#5035:UpdateRemotePluginsHere are the headline features of the previous releases; for details see the release notes.
These screenshots demo what’s possible. They may use unreleased features, third-party UIs, plugins, or config changes.
Graphical interfaces are not packaged with Neovim. GUIs are not limited by terminal capabilities and can choose to override the rendering of certain neovim elements (cursor, tabline, popup menu, etc). This section showcases only a handful of them, see Neovim’s wiki for a list of many of the available GUIs.
We don't have an "administrative" staff. Funding goes directly to software development.
Email: admin@neovim.io
Open Source Collective 501(c)(6)
EIN: 82-2037583
440 N Barranca Ave #3939 Covina, CA 91723 United States
hello@oscollective.org
Details
You can also donate via GitHub Sponsors, which will be routed to OpenCollective.
Funding makes it possible for core developers to work full-time for a month or longer, accelerating projects like Lua stdlib, treesitter parser engine, LSP framework, extended marks, embedded terminal, job control, RPC API, and remote UIs.
Neovim is a refactor, and sometimes redactor, in the tradition of Vim (which itself derives from Stevie). It is not a rewrite but a continuation and extension of Vim. Many clones and derivatives exist, some very clever—but none are Vim. Neovim is built for users who want the good parts of Vim, and more.
Extensible. Usable. Vim.
In the 1980s, if you wanted your computer to do floating-point calculations faster, you could buy the Intel 8087 floating-point coprocessor chip. Plugging it into your IBM PC would make operations up to 100 times faster, a big boost for spreadsheets and other number-crunching applications. The 8087 uses complicated algorithms to compute trigonometric, logarithmic, and exponential functions. These algorithms are implemented inside the chip in microcode. I'm part of a group that is reverse-engineering this microcode. In this post, I examine the 49 types of conditional tests that the 8087's microcode uses inside its algorithms. Some conditions are simple, such as checking if a number is zero or negative, while others are specialized, such as determining what direction to round a number.
To explore the 8087's circuitry, I opened up an 8087 chip and took numerous photos of the silicon die with a microscope. Around the edges of the die, you can see the hair-thin bond wires that connect the chip to its 40 external pins. The complex patterns on the die are formed by its metal wiring, as well as the polysilicon and silicon underneath. The bottom half of the chip is the "datapath", the circuitry that performs calculations on 80-bit floating point values. At the left of the datapath, a constant ROM holds important constants such as π. At the right are the eight registers that the programmer uses to hold floating-point values; in an unusual design decision, these registers are arranged as a stack.
The chip's instructions are defined by the large microcode ROM in the middle. To execute a floating-point instruction, the 8087 decodes the instruction and the microcode engine starts executing the appropriate micro-instructions from the microcode ROM. The microcode decode circuitry to the right of the ROM generates the appropriate control signals from each micro-instruction.1 The bus registers and control circuitry handle interactions with the main 8086 processor and the rest of the system.
Executing an 8087 instruction such as arctan requires hundreds of internal steps to compute the result. These steps are implemented in microcode with micro-instructions specifying each step of the algorithm. (Keep in mind the difference between the assembly language instructions used by a programmer and the undocumented low-level micro-instructions used internally by the chip.) The microcode ROM holds 1648 micro-instructions, implementing the 8087's instruction set. Each micro-instruction is 16 bits long and performs a simple operation such as moving data inside the chip, adding two values, or shifting data. I'm working with the "Opcode Collective" to reverse engineer the micro-instructions and fully understand the microcode (link).
The microcode engine (below) controls the execution of micro-instructions, acting as the mini-CPU inside the 8087. Specifically, it generates an 11-bit micro-address, the address of a micro-instruction in the ROM. The microcode engine implements jumps, subroutine calls, and returns within the microcode. These jumps, subroutine calls, and returns are all conditional; the microcode engine will either perform the operation or skip it, depending on the value of a specified condition.
I'll write more about the microcode engine later, but I'll give an overview here. At the top, the Instruction Decode PLA2 decodes an 8087 instruction to determine the starting address in microcode. Below that, the Jump PLA holds microcode addresses for jumps and subroutine calls. Below this, six 11-bit registers implement the microcode stack, allowing six levels of subroutine calls inside the microcode. (Note that this stack is completely different from the 8087's register stack that holds eight floating-point values.) The stack registers have associated read/write circuitry. The incrementer adds one to the micro-address to step through the code. The engine also implements relative jumps, using an adder to add an offset to the current location. At the bottom, the address latch and drivers boost the 11-bit address output and send it to the microcode ROM.
A micro-instruction can say "jump ahead 5 micro-instructions if a register is zero" and the microcode engine will either perform the jump or ignore it, based on the register value. In the circuitry, the condition causes the microcode engine to either perform the jump or block the jump. But how does the hardware select one condition out of the large set of conditions?
Six bits of the micro-instruction can specify one of 64 conditions. A circuit similar to the idealized diagram below selects the specified condition. The key component is a multiplexer, represented by a trapezoid below. A multiplexer is a simple circuit that selects one of its four inputs. By arranging multiplexers in a tree, one of the 64 conditions on the left is selected and becomes the output, passed to the microcode engine.
For example, if bits J and K of the microcode are 00, the rightmost multiplexer will select the first input. If bits LM are 01, the middle multiplexer will select the second input, and if bits NO are 10, the left multiplexer will select its third input. The result is that condition 06 will pass through the tree and become the output.3 By changing the bits that control the multiplexers, any of the inputs can be used. (We've arbitrarily given the 16 microcode bits the letter names A through P.)
Physically, the conditions come from locations scattered across the die. For instance, conditions involving the opcode come from the instruction decoding part of the chip, while conditions involving a register are evaluated next to the register. It would be inefficient to run 64 wires for all the conditions to the microcode engine. The tree-based approach reduces the wiring since the "leaf" multiplexers can be located near the associated condition circuitry. Thus, only one wire needs to travel a long distance rather than multiple wires. In other words, the condition selection circuitry is distributed across the chip instead of being implemented as a centralized module.
Because the conditions don't always fall into groups of four, the actual implementation is slightly different from the idealized diagram above. In particular, the top-level multiplexer has five inputs, rather than four.4 Other multiplexers don't use all four inputs. This provides a better match between the physical locations of the condition circuits and the multiplexers. In total, 49 of the possible 64 conditions are implemented in the 8087.
The circuit that selects one of the four conditions is called a multiplexer. It is constructed from pass transistors, transistors that are configured to either pass a signal through or block it. To operate the multiplexer, one of the select lines is energized, turning on the corresponding pass transistor. This allows the selected input to pass through the transistor to the output, while the other inputs are blocked.
The diagram below shows how a multiplexer appears on the die. The pinkish regions are doped silicon. The white lines are polysilicon wires. When polysilicon crosses over doped silicon, a transistor is formed. On the left is a four-way multiplexer, constructed from four pass transistors. It takes inputs (black) for four conditions, numbered 38, 39, 3a, and 3b. There are four control signals (red) corresponding to the four combinations of bits N and O. One of the inputs will pass through a transistor to the output, selected by the active control signal. The right half contains the logic (four NOR gates and two inverters) to generate the control signals from the microcode bits. (Metal lines run horizontally from the logic to the control signal contacts, but I dissolved the metal for this photo.) Each multiplexer in the 8087 has a completely different layout, manually optimized based on the location of the signals and surrounding circuitry. Although the circuit for a multiplexer is regular (four transistors in parallel), the physical layout looks somewhat chaotic.
The 8087 uses pass transistors for many circuits, not just multiplexers. Circuits with pass transistors are different from regular logic gates because the pass transistors provide no amplification. Instead, signals get weaker as they go through pass transistors. To solve this problem, inverters or buffers are inserted into the condition tree to boost signals; they are omitted from the diagram above.
Of the 8087's 49 different conditions, some are widely used in the microcode, while others are designed for a specific purpose and are only used once. The full set of conditions is described in a footnote7 but I'll give some highlights here.
Fifteen conditions examine the bits of the current instruction's opcode. This allows one microcode routine to handle a group of similar instructions and then change behavior based on the specific instruction. For example, conditions test if the instruction is multiplication, if the instruction is an FILD/FIST (integer load or store), or if the bottom bit of the opcode is set.5
The 8087 has three temporary registers—tmpA, tmpB, and tmpC—that hold values during computation. Various conditions examine the values in the tmpA and tmpB registers.6 In particular, the 8087 uses an interesting way to store numbers internally: each 80-bit floating-point value also has two "tag" bits. These bits are mostly invisible to the programmer and can be thought of as metadata. The tag bits indicate if a register is empty, contains zero, contains a "normal" number, or contains a special value such as NaN (Not a Number) or infinity. The 8087 uses the tag bits to optimize operations. The tags also detect stack overflow (storing to a non-empty stack register) or stack underflow (reading from an empty stack register).
Other conditions are highly specialized. For instance, one condition looks at the rounding mode setting and the sign of the value to determine if the value should be rounded up or down. Other conditions deal with exceptions such as numbers that are too small (i.e. denormalized) or numbers that lose precision. Another condition tests if two values have the same sign or not. Yet another condition tests if two values have the same sign or not, but inverts the result if the current instruction is subtraction. The simplest condition is simply "true", allowing an unconditional branch.
For flexibility, conditions can be "flipped", either jumping if the condition is true or jumping if the condition is false. This is controlled by bit P of the microcode. In the circuitry, this is implemented by a gate that XORs the P bit with the condition. The result is that the state of the condition is flipped if bit P is set.
For a concrete example of how conditions are used, consider the
microcode routine
that implements FCHS and FABS, the
instructions to change the sign and compute the absolute value, respectively.
These operations are almost the same (toggling the sign bit versus clearing the sign bit), so the same
microcode routine handles both instructions, with a jump instruction to handle the difference.
The FABS and FCHS instructions were designed with identical opcodes,
except that the bottom bit is set for FABS.
Thus, the microcode routine uses a condition that tests the bottom bit, allowing the routine to branch and
change its behavior for FABS vs FCHS.
Looking at the relevant micro-instruction, it has the hex value
0xc094, or in binary 110 000001 001010 0.
The first three bits (ABC=110) specify the relative jump operation (100 would jump to a fixed target and 101 would
perform a subroutine call.)
Bits D through I (000010) indicate the amount of the jump (+`).
Bits J through O (001010, hex 0a) specify the condition to test, in this case, the last bit of the instruction opcode.
The final bit (P) would toggle the condition if set, (i.e. jump if false).
Thus, for FABS, the jump instruction will jump ahead one micro-instruction.
This has the effect of skipping the next micro-instruction, which sets the appropriate sign bit for
FCHS.
The 8087 performs floating-point operations much faster than the 8086 by using special hardware, optimized for floating-point. The condition code circuitry is one example of this: the 8087 can test a complicated condition in a single operation. However, these complicated conditions make it much harder to understand the microcode. But by a combination of examining the circuitry and looking at the micocode, we're making progress. Thanks to the members of the "Opcode Collective" for their hard work, especially Smartest Blob and Gloriouscow.
For updates, follow me on Bluesky (@righto.com), Mastodon (@kenshirriff@oldbytes.space), or RSS.
The section of the die that I've labeled "Microcode decode" performs some of the microcode decoding, but large parts of the decoding are scattered across the chip, close to the circuitry that needs the signals. This makes reverse-engineering the microcode much more difficult. I thought that understanding the microcode would be straightforward, just examining a block of decode circuitry. But this project turned out to be much more complicated and I need to reverse-engineer the entire chip. ↩
A PLA is a "Programmable Logic Array". It is a technique to implement logic functions with grids of transistors. A PLA can be used as a compressed ROM, holding data in a more compact representation. (Saving space was very important in chips of this era.) In the 8087, PLAs are used to hold tables of microcode addresses. ↩
Note that the multiplexer circuit selects the condition corresponding to the binary value of the bits. In the example, bits 000110 (0x06) select condition 06. ↩
The five top-level multiplexer inputs correspond to bit patterns 00, 011, 10, 110, and 111. That is, two inputs depend on bits J and K, while three inputs depend on bits J, K, and L. The bit pattern 010 is unused, corresponding to conditions 0x10 through 0x17, which aren't implemented. ↩
The 8087 acts as a co-processor with the 8086 processor. The 8086 instruction set is designed so instructions with a special "ESCAPE" sequence in the top 5 bits are processed by the co-processor, in this case the 8087. Thus, the 8087 receives a 16-bit instruction, but only the bottom 11 bits are usable. For a memory operation, the second byte of the instruction is an 8086-style ModR/M byte. For instructions that don't access memory, the second byte specifies more of the instruction and sometimes specifies the stack register to use for the instruction.
The relevance of this is that the 8087's microcode engine uses the 11 bits of the instruction to determine which microcode routine to execute. The microcode also uses various condition codes to change behavior depending on different bits of the instruction. ↩
There is a complication with the tmpA and tmpB registers: they can be swapped with the micro-instruction "ABC.EF". The motivation behind this is that if you have two arguments, you can use a micro-subroutine to load an argument into tmpA, swap the registers, and then use the same subroutine to load the second argument into tmpA. The result is that the two arguments end up in tmpB and tmpA without any special coding in the subroutine.
The implementation doesn't physically swap the registers, but renames them internally, which is much more efficient. A flip-flop is toggled every time the registers are swapped. If the flip-flop is set, a request goes to one register, while if the flip-flop is clear, a request goes to the other register. (Many processors use the same trick. For instance, the Intel 8080 has an instruction to exchange the DE and HL registers. The Z80 has an instruction to swap register banks. In both cases, a flip-flop renames the registers, so the data doesn't need to move.) ↩
The table below is the real meat of this post, the result of much circuit analysis. These details probably aren't interesting to most people, so I've relegated the table to a footnote. Descriptions in italics are provided by Smartest Blob based on examination of the microcode. Grayed-out lines are unused conditions.
The table has five sections, corresponding to the 5 inputs to the top-level condition multiplexer. These inputs come from different parts of the chip, so the sections correspond to different categories of conditions.
The first section consists of instruction parsing, with circuitry near the microcode engine. The description shows the 11-bit opcode pattern that triggers the condition, with 0 bits and 1 bits as specified, and X indicating a "don't care" bit that can be 0 or 1. Where simpler, I list the relevant instructions instead.
The next section indicates conditions on the exponent. I am still investigating these conditions, so the descriptions are incomplete. The third section is conditions on the temporary registers or conditions related to the control register. These circuits are to the right of the microcode ROM.
Conditions in the fourth section examine the floating-point bus, with circuitry near the bottom of the chip. Conditions 34 and 35 use a special 16-bit bidirectional shift register, at the far right of the chip. The top bit from the floating-point bus is shifted in. Maybe this shift register is used for CORDIC calculations? The conditions in the final block are miscellaneous, including the always-true condition 3e, which is used for unconditional jumps.
| Cond. | Description |
|---|---|
| 00 | not XXX 11XXXXXX |
| 01 | 1XX 11XXXXXX |
| 02 | 0XX 11XXXXXX |
| 03 | X0X XXXXXXXX |
| 04 | not cond 07 or 1XX XXXXXXXX |
| 05 | not FLD/FSTP temp-real or BCD |
| 06 | 110 xxxxxxxx or 111 xx0xxxxx |
| 07 | FLD/FSTP temp-real |
| 08 | FBLD/FBSTP |
| 09 | |
| 0a | XXX XXXXXXX1 |
| 0b | XXX XXXX1XXX |
| 0c | FMUL |
| 0d | FDIV FDIVR |
| 0e | FADD FCOM FCOMP FCOMPP FDIV FDIVR FFREE FLD FMUL FST FSTP FSUB FSUBR FXCH |
| 0f | FCOM FCOMP FCOMPP FTST |
| 10 | |
| 11 | |
| 12 | |
| 13 | |
| 14 | |
| 15 | |
| 16 | |
| 17 | |
| 18 | exponent condition |
| 19 | exponent condition |
| 1a | exponent condition |
| 1b | exponent condition |
| 1c | exponent condition |
| 1d | exponent condition |
| 1e | eight exponent zero bits |
| 1f | exponent condition |
| 20 | tmpA tag ZERO |
| 21 | tmpA tag SPECIAL |
| 22 | tmpA tag VALID |
| 23 | stack overflow |
| 24 | tmpB tag ZERO |
| 25 | tmpB tag SPECIAL |
| 26 | tmpB tag VALID |
| 27 | st(i) doesn't exist (A)? |
| 28 | tmpA sign |
| 29 | tmpB top bit |
| 2a | tmpA zero |
| 2b | tmpA top bit |
| 2c | Control Reg bit 12: infinity control |
| 2d | round up/down |
| 2e | unmasked interrupt |
| 2f | DE (denormalized) interrupt |
| 30 | top reg bit |
| 31 | |
| 32 | reg bit 64 |
| 33 | reg bit 63 |
| 34 | Shifted top bits, all zero |
| 35 | Shifted top bits, one out |
| 36 | |
| 37 | |
| 38 | const latch zero |
| 39 | tmpA vs tmpB sign, flipped for subtraction |
| 3a | precision exception |
| 3b | tmpA vs tmpB sign |
| 3c | |
| 3d | |
| 3e | unconditional |
| 3f |
This table is under development and undoubtedly has errors. ↩
Early microprocessors were very slow when operating with floating-point numbers. But in 1980, Intel introduced the 8087 floating-point coprocessor, performing floating-point operations up to 100 times faster. This was a huge benefit for IBM PC applications such as AutoCAD, spreadsheets, and flight simulators. The 8087 was so effective that today's computers still use a floating-point system based on the 8087.1
The 8087 was an extremely complex chip for its time, containing somewhere between 40,000 and 75,000 transistors, depending on the source.2 To explore how the 8087 works, I opened up a chip and took numerous photos of the silicon die with a microscope. Around the edges of the die, you can see the hair-thin bond wires that connect the chip to its 40 external pins. The complex patterns on the die are formed by its metal wiring, as well as the polysilicon and silicon underneath. The bottom half of the chip is the "datapath", the circuitry that performs calculations on 80-bit floating point values. At the left of the datapath, a constant ROM holds important constants such as π. At the right are the eight registers that form the stack, along with the stack control circuitry.
The chip's instructions are defined by the large microcode ROM in the middle. This ROM is very unusual; it is semi-analog, storing two bits per transistor by using four transistor sizes. To execute a floating-point instruction, the 8087 decodes the instruction and the microcode engine starts executing the appropriate micro-instructions from the microcode ROM. The decode circuitry to the right of the ROM generates the appropriate control signals from each micro-instruction. The bus registers and control circuitry handle interactions with the main 8086 processor and the rest of the system. Finally, the bias generator uses a charge pump to create a negative voltage to bias the chip's substrate, the underlying silicon.
The stack registers and control circuitry (in red above) are the subject of this blog post. Unlike most processors, the 8087 organizes its registers in a stack, with instructions operating on the top of the stack. For instance, the square root instruction replaces the value on the top of the stack with its square root. You can also access a register relative to the top of the stack, for instance, adding the top value to the value two positions down from the top. The stack-based architecture was intended to improve the instruction set, simplify compiler design, and make function calls more efficient, although it didn't work as well as hoped.
The diagram above shows how the stack operates. The stack consists of eight registers, with the Stack Top (ST) indicating the current top of the stack. To push a floating-point value onto the stack, the Stack Top is decremented and then the value is stored in the new top register. A pop is performed by copying the value from the stack top and then incrementing the Stack Top. In comparison, most processors specify registers directly, so register 2 is always the same register.
The stack registers occupy a substantial area on the die of the 8087 because floating-point numbers take many bits. A floating-point number consists of a fractional part (sometimes called the mantissa or significand), along with the exponent part; the exponent allows floating-point numbers to cover a range from extremely small to extremely large. In the 8087, floating-point numbers are 80 bits: 64 bits of significand, 15 bits of exponent, and a sign bit. An 80-bit register was very large in the era of 8-bit or 16-bit computers; the eight registers in the 8087 would be equivalent to 40 registers in the 8086 processor.
The registers store each bit in a static RAM cell. Each cell has two inverters connected in a loop. This circuit forms a stable feedback loop, with one inverter on and one inverter off. Depending on which inverter is on, the circuit stores a 0 or a 1. To write a new value into the circuit, one of the lines is pulled low, flipping the loop into the desired state. The trick is that each inverter uses a very weak transistor to pull the output high, so its output is easily overpowered to change the state.
These inverter pairs are arranged in an 8 × 80 grid that implements eight words of 80 bits. Each of the 80 rows has two bitlines that provide access to a bit. The bitlines provide both read and write access to a bit; the pair of bitlines allows either inverter to be pulled low to store the desired bit value. Eight vertical wordlines enable access to one word, one column of 80 bits. Each wordline turns on 160 pass transistors, connecting the bitlines to the inverters in the selected column. Thus, when a wordline is enabled, the bitlines can be used to read or write that word.
Although the chip looks two-dimensional, it actually consists of multiple layers. The bottom layer is silicon. The pinkish regions below are where the silicon has been "doped" to change its electrical properties, making it an active part of the circuit. The doped silicon forms a grid of horizontal and vertical wiring, with larger doped regions in the middle. On top of the silicon, polysilicon wiring provides two functions. First, it provides a layer of wiring to connect the circuit. But more importantly, when polysilicon crosses doped silicon, it forms a transistor. The polysilicon provides the gate, turning the transistor on and off. In this photo, the polysilicon is barely visible, so I've highlighted part of it in red. Finally, horizontal metal wires provide a third layer of interconnecting wiring. Normally, the metal hides the underlying circuitry, so I removed the metal with acid for this photo. I've drawn blue lines to represent the metal layer. Contacts provide connections between the various layers.
The layers combine to form the inverters and selection transistors of a memory cell, indicated with the dotted line below. There are six transistors (yellow), where polysilicon crosses doped silicon. Each inverter has a transistor that pulls the output low and a weak transistor to pull the output high. When the word line (vertical polysilicon) is active, it connects the selected inverters to the bit lines (horizontal metal) through the two selection transistors. This allows the bit to be read or written.
Each register has two tag bits associated with it, an unusual form of metadata to indicate if the register is empty, contains zero, contains a valid value, or contains a special value such as infinity. The tag bits are used to optimize performance internally and are mostly irrelevant to the programmer. As well as being accessed with a register, the tag bits can be accessed in parallel as a 16-bit "Tag Word". This allows the tags to be saved or loaded as part of the 8087's state, for instance, during interrupt handling.
The decoder circuit, wedged into the middle of the register file, selects one of the registers. A register is specified internally with a 3-bit value. The decoder circuit energizes one of the eight register select lines based on this value.
The decoder circuitry is straightforward: it has eight 3-input NOR gates to match one of the eight bit patterns. The select line is then powered through a high-current driver that uses large transistors. (In the photo below, you can compare the large serpentine driver transistors to the small transistors in a bit cell.)
The decoder has an interesting electrical optimization. As shown earlier, the register select lines are eight polysilicon lines running vertically, the length of the register file. Unfortunately, polysilicon has fairly high resistance, better than silicon but much worse than metal. The problem is that the resistance of a long polysilicon line will slow down the system. That is, the capacitance of transistor gates in combination with high resistance causes an RC (resistive-capacitive) delay in the signal.
The solution is that the register select lines also run in the metal layer, a second set of lines immediately to the right of the register file. These lines branch off from the register file about 1/3 of the way down, run to the bottom, and then connect back to the polysilicon select lines at the bottom. This reduces the maximum resistance through a select line, increasing the speed.
A stack needs more control circuitry than a regular register file, since the circuitry must keep track of the position of the top of the stack.3 The control circuitry increments and decrements the top of stack (TOS) pointer as values are pushed or popped (purple).4 Moreover, an 8087 instruction can access a register based on its offset, for instance the third register from the top. To support this, the control circuitry can temporarily add an offset to the top of stack position (green). A multiplexer (red) selects either the top of stack or the adder output, and feeds it to the decoder (blue), which selects one of the eight stack registers in the register file (yellow), as described earlier.
The physical implementation of the stack circuitry is shown below. The logic at the top selects the stack operation based on the 16-bit micro-instruction.5 Below that are the three latches that hold the top of stack value. (The large white squares look important, but they are simply "jumpers" from the ground line to the circuitry, passing under metal wires.)
The three-bit adder is at the bottom, along with the multiplexer. You might expect the adder to use a simple "full adder" circuit. Instead, it is a faster carry-lookahead adder. I won't go into details here, but the summary is that at each bit position, an AND gate produces a Carry Generate signal while an XOR gate produces a Carry Propagate signal. Logic gates combine these signals to produce the output bits in parallel, avoiding the slowdown of the carry rippling through the bits.
The incrementer/decrementer uses a completely different approach. Each of the three bits uses a toggle flip-flop. A few logic gates determine if each bit should be toggled or should keep its previous value. For instance, when incrementing, the top bit is toggled if the lower bits are 11 (e.g. incrementing from 011 to 100). For decrementing, the top bit is toggled if the lower bits are 00 (e.g. 100 to 011). Simpler logic determines if the middle bit should be toggled. The bottom bit is easier, toggling every time whether incrementing or decrementing.
The schematic below shows the circuitry for one bit of the stack. Each bit is implemented with a moderately complicated flip-flop that can be cleared, loaded with a value, or toggled, based on control signals from the microcode. The flip-flop is constructed from two set-reset (SR) latches. Note that the flip-flop outputs are crossed when fed back to the input, providing the inversion for the toggle action. At the right, the multiplexer selects either the register value or the sum from the adder (not shown), generating the signals to the decoder.
According to the designers of the 8087,7 the main motivation for using a stack rather than a flat register set was that instructions didn't have enough bits to address multiple register operands. In addition, a stack has "advantages over general registers for expression parsing and nested function calls." That is, a stack works well for a mathematical expression since sub-expressions can be evaluated on the top of the stack. And for function calls, you avoid the cost of saving registers to memory, since the subroutine can use the stack without disturbing the values underneath. At least that was the idea.
The main problem is "stack overflow". The 8087's stack has eight entries, so if you push a ninth value onto the stack, the stack will overflow. Specifically, the top-of-stack pointer will wrap around, obliterating the bottom value on the stack. The 8087 is designed to detect a stack overflow using the register tags: pushing a value to a non-empty register triggers an invalid operation exception.6
The designers expected that stack overflow would be rare and could be handled by the operating system (or library code). After detecting a stack overflow, the software should dump the existing stack to memory to provide the illusion of an infinite stack. Unfortunately, bad design decisions made it difficult "both technically and commercially" to handle stack overflow.
One of the 8087's designers (Kahan) attributes the 8087's stack problems to the time difference between California, where the designers lived, and Israel, where the 8087 was implemented. Due to a lack of communication, each team thought the other was implementing the overflow software. It wasn't until the 8087 was in production that they realized that "it might not be possible to handle 8087 stack underflow/overflow in a reasonable way. It's not impossible, just impossible to do it in a reasonable way."
As a result, the stack was largely a problem rather than a solution.
Most 8087 software saved the full stack to memory before performing
a function call, creating more memory traffic.
Moreover, compilers turned out to work better with regular registers than a stack,
so compiler writers awkwardly used the stack to emulate regular registers.
The GCC compiler reportedly needs 3000 lines of extra code to support the x87 stack.
In the 1990s, Intel introduced a new floating-point system called SSE, followed by AVX in 2011. These systems use regular (non-stack) registers and provide parallel operations for higher performance, making the 8087's stack instructions largely obsolete.
At the start, Intel was unenthusiastic about producing the 8087, viewing it as unlikely to be a success. John Palmar, a principal architect of the chip, had little success convincing skeptical Intel management that the market for the 8087 was enormous. Eventually, he said, "I'll tell you what. I'll relinquish my salary, provided you'll write down your number of how many you expect to sell, then give me a dollar for every one you sell beyond that."7 Intel didn't agree to the deal—which would have made a fortune for Palmer—but they reluctantly agreed to produce the chip.
Intel's Santa Clara engineers shunned the 8087, considering it unlikely to work: the 8087 would be two to three times more complex than the 8086, with a die so large that a wafer might not have a single working die. Instead, Rafi Nave, at Intel's Israel site, took on the risky project: “Listen, everybody knows it's not going to work, so if it won't work, I would just fulfill their expectations or their assessment. If, by chance, it works, okay, then we'll gain tremendous respect and tremendous breakthrough on our abilities.”
A small team of seven engineers developed the 8087 in Israel. They designed the chip on Mylar sheets: a millimeter on Mylar represented a micron on the physical chip. The drawings were then digitized on a Calma system by clicking on each polygon to create the layout. When the chip was moved into production, the yield was very low but better than feared: two working dies per four-inch wafer.
The 8087 ended up being a large success, said to have been Intel's most profitable product line at times. The success of the 8087 (along with the 8088) cemented the reputation of Intel Israel, which eventually became Israel's largest tech employer. The benefits of floating-point hardware proved to be so great that Intel integrated the floating-point unit into later processors starting with the 80486 (1989). Nowadays, most modern computers, from cellphones to mainframes, provide floating point based on the 8087, so I consider the 8087 one of the most influential chips ever created.
For more, follow me on Bluesky (@righto.com), Mastodon (@kenshirriff@oldbytes.space), or RSS. I wrote some articles about the 8087 a few years ago, including the die, the ROM, the bit shifter, and the constants, so you may have seen some of this material before.
Most computers now use the IEEE 754 floating-point standard, which is based on the 8087. This standard has been awarded a milestone in computation. ↩
Curiously, reliable sources differ on the number of transistors in the 8087 by almost a factor of 2. Intel says 40,000, as does designer William Kahan (link). But in A Numeric Data Processor, designers Rafi Nave and John Palmer wrote that the chip contains "the equivalent of over 65,000 devices" (whatever "equivalent" means). This number is echoed by a contemporary article in Electronics (1980) that says "over 65,000 H-MOS transistors on a 78,000-mil2 die." Many other sources, such as Upgrading & Repairing PCs, specify 45,000 transistors. Designer Rafi Nave stated that the 8087 has 63,000 or 64,000 transistors if you count the ROM transistors directly, but if you count ROM transistors as equivalent to two transistors, then you get about 75,000 transistors. ↩
The 8087 has a 16-bit Status Word that contains the stack top pointer, exception flags, the four-bit condition code, and other values. Although the Status Word appears to be a 16-bit register, it is not implemented as a register. Instead, parts of the Status Word are stored in various places around the chip: the stack top pointer is in the stack circuitry, the exception flags are part of the interrupt circuitry, the condition code bits are next to the datapath, and so on. When the Status Word is read or written, these various circuits are connected to the 8087's internal data bus, making the Status Word appear to be a monolithic entity. Thus, the stack circuitry includes support for reading and writing it. ↩
Intel filed several patents on the 8087, including Numeric data processor, another Numeric data processor, Programmable bidirectional shifter, Fraction bus for use in a numeric data processor, and System bus arbitration, circuitry and methodology. ↩
I started looking at the stack in detail to reverse engineer the micro-instruction format and determine how the 8087's microcode works. I'm working with the "Opcode Collective" on Discord on this project, but progress is slow due to the complexity of the micro-instructions. ↩
The 8087 detects stack underflow in a similar manner. If you pop more values from the stack than are present, the tag will indicate that the register is empty and shouldn't be accessed. This triggers an invalid operation exception. ↩
The 8087 is described in detail in The 8086 Family User's Manual, Numerics Supplement. An overview of the stack is on page 60 of The 8087 Primer by Palmer and Morse. More details are in Kahan's On the Advantages of the 8087's Stack, an unpublished course note (maybe for CS 279?) with a date of Nov 2, 1990 or perhaps August 23, 1994. Kahan discusses why the 8087's design makes it hard to handle stack overflow in How important is numerical accuracy, Dr. Dobbs, Nov. 1997. Another information source is the Oral History of Rafi Nave ↩↩
I've been studying the standard cell circuitry in the Intel 386 processor recently. The 386, introduced in 1985, was Intel's most complex processor at the time, containing 285,000 transistors. Intel's existing design techniques couldn't handle this complexity and the chip began to fall behind schedule. To meet the schedule, the 386 team started using a technique called standard cell logic. Instead of laying out each transistor manually, the layout process was performed by a computer.
The idea behind standard cell logic is to create standardized circuits (standard cells) for each type of logic element, such as an inverter, NAND gate, or latch. You feed your circuit description into software that selects the necessary cells, positions these cells into columns, and then routes the wiring between the cells. This "automatic place and route" process creates the chip layout much faster than manual layout. However, switching to standard cells was a risky decision since if the software couldn't create a dense enough layout, the chip couldn't be manufactured. But in the end, the 386 finished ahead of schedule, an almost unheard-of accomplishment.1
The 386's standard cell circuitry contains a few circuits that I didn't expect. In this blog post, I'll take a quick look at some of these circuits: surprisingly large multiplexers, a transistor that doesn't fit into the standard cell layout, and inverters that turned out not to be inverters. (If you want more background on standard cells in the 386, see my earlier post, "Reverse engineering standard cell logic in the Intel 386 processor".)
The photo below shows the 386 die with the automatic-place-and-route regions highlighted; I'm focusing on the red region in the lower right. These blocks of logic have cells arranged in rows, giving them a characteristic striped appearance. The dark stripes are the transistors that make up the logic gates, while the lighter regions between the stripes are the "routing channels" that hold the wiring that connects the cells. In comparison, functional blocks such as the datapath on the left and the microcode ROM in the lower right were designed manually to optimize density and performance, giving them a more solid appearance.
As for other features on the chip, the black circles around the border are bond wire connections that go to the chip's external pins. The chip has two metal layers, a small number by modern standards, but a jump from the single metal layer of earlier processors such as the 286. (Providing two layers of metal made automated routing practical: one layer can hold horizontal wires while the other layer can hold vertical wires.) The metal appears white in larger areas, but purplish where circuitry underneath roughens its surface. The underlying silicon and the polysilicon wiring are obscured by the metal layers.
The standard cell circuitry that I'm examining (red box above) is part of the control logic that selects registers while executing an instruction. You might think that it is easy to select which registers take part in an instruction, but due to the complexity of the x86 architecture, it is more difficult. One problem is that a 32-bit register such as EAX can also be treated as the 16-bit register AX, or two 8-bit registers AH and AL. A second problem is that some instructions include a "direction" bit that switches the source and destination registers. Moreover, sometimes the register is specified by bits in the instruction, but in other cases, the register is specified by the microcode. Due to these factors, selecting the registers for an operation is a complicated process with many cases, using control bits from the instruction, from the microcode, and from other sources.
Three registers need to be selected for an operation—two source registers and a destination register—and there are about 17 cases that need to be handled. Registers are specified with 7-bit control signals that select one of the 30 registers and control which part of the register is accessed. With three control signals, each 7 bits wide, and about 17 cases for each, you can see that the register control logic is large and complicated. (I wrote more about the 386's registers here.)
I'm still reverse engineering the register control logic, so I won't go into details. Instead, I'll discuss how the register control circuit uses multiplexers, implemented with standard cells. A multiplexer is a circuit that combines multiple input signals into a single output by selecting one of the inputs.2 A multiplexer can be implemented with logic gates, for instance, by ANDing each input with the corresponding control line, and then ORing the results together. However, the 386 uses a different approach—CMOS switches—that avoids a large AND/OR gate.
The schematic above shows how a CMOS switch is constructed from two MOS transistors. When the two transistors are on, the output is connected to the input, but when the two transistors are off, the output is isolated. An NMOS transistor is turned on when its input is high, but a PMOS transistor is turned on when its input is low. Thus, the switch uses two control inputs, one inverted. The motivation for using two transistors is that an NMOS transistor is better at pulling the output low, while a PMOS transistor is better at pulling the output high, so combining them yields the best performance.3 Unlike a logic gate, the CMOS switch has no amplification, so a signal is weakened as it passes through the switch. As will be seen below, inverters can be used to amplify the signal.
The image below shows how CMOS switches appear under the microscope. This image is very hard to interpret because the two layers of metal on the 386 are packed together densely, but you can see that some wires run horizontally and others run vertically. The bottom layer of metal (called M1) runs vertically in the routing area, as well as providing internal wiring for a cell. The top layer of metal (M2) runs horizontally; unlike M1, the M2 wires can cross a cell. The large circles are vias that connect the M1 and M2 layers, while the small circles are connections between M1 and polysilicon or M1 and silicon. The central third of the image is a column of standard cells with two CMOS switches outlined in green. The cells are bordered by the vertical ground rail and +5V rail that power the cells. The routing areas are on either side of the cells, holding the wiring that connects the cells.
Removing the metal layers reveals the underlying silicon with a layer of polysilicon wiring on top. The doped silicon regions show up as dark outlines. I've drawn the polysilicon in green; it forms a transistor (brighter green) when it crosses doped silicon. The metal ground and power lines are shown in blue and red, respectively, with other metal wiring in purple. The black dots are vias between layers. Note how metal wiring (purple) and polysilicon wiring (green) are combined to route signals within the cell. Although this standard cell is complicated, the important thing is that it only needs to be designed once. The standard cells for different functions are all designed to have the same width, so the cells can be arranged in columns, snapped together like Lego bricks.
To summarize, this switch circuit allows the input to be connected to the output or disconnected, controlled by the select signal. This switch is more complicated than the earlier schematic because it includes two inverters to amplify the signal. The data input and the two select lines are connected to the polysilicon (green); the cell is designed so these connections can be made on either side. At the top, the input goes through a standard two-transistor inverter. The lower left has two transistors, combining the NMOS half of an inverter with the NMOS half of the switch. A similar circuit on the right combines the PMOS part of an inverter and switch. However, because PMOS transistors are weaker, this part of the circuit is duplicated.
A multiplexer is constructed by combining multiple switches, one for each input. Turning on one switch will select the corresponding input. For instance, a four-to-one multiplexer has four switches, so it can select one of the four inputs.
The schematic above shows a hypothetical multiplexer with four inputs. One optimization is that if an input is always 0, the PMOS transistor can be omitted. Likewise, if an input is always 1, the NMOS transistor can be omitted. One set of select lines is activated at a time to select the corresponding input. The pink circuit selects 1, green selects input A, yellow selects input B, and blue selects 0. The multiplexers in the 386 are similar, but have more inputs.
The diagram below shows how much circuitry is devoted to multiplexers in this block of standard cells. The green, purple, and red cells correspond to the multiplexers driving the three register control outputs. The yellow cells are inverters that generate the inverted control signals for the CMOS switches. This diagram also shows how the automatic layout of cells results in a layout that appears random.
The idea of standard-cell logic is that standardized cells are arranged in columns. The space between the cells is the "routing channel", holding the wiring that links the cells. The 386 circuitry follows this layout, except for one single transistor, sitting between two columns of cells.
I wrote some software tools to help me analyze the standard cells. Unfortunately, my tools assumed that all the cells were in columns, so this one wayward transistor caused me considerable inconvenience.
The transistor turns out to be a PMOS transistor, pulling a signal high as part of a multiplexer. But why is this transistor out of place? My hypothesis is that the transistor is a bug fix. Regenerating the cell layout was very costly, taking many hours on an IBM mainframe computer. Presumably, someone found that they could just stick the necessary transistor into an unused spot in the routing channel, manually add the necessary wiring, and avoid the delay of regenerating all the cells.
The simplest CMOS gate is the inverter, with an NMOS transistor to pull the output low and a PMOS transistor to pull the output high. The standard cell circuitry that I examined contains over a hundred inverters of various sizes. (Performance is improved by using inverters that aren't too small but also aren't larger than necessary for a particular circuit. Thus, the standard cell library includes inverters of multiple sizes.)
The image below shows a medium-sized standard-cell inverter under the microscope. For this image, I removed the two metal layers with acid to show the underlying polysilicon (bright green) and silicon (gray). The quality of this image is poor—it is difficult to remove the metal without destroying the polysilicon—but the diagram below should clarify the circuit. The inverter has two transistors: a PMOS transistor connected to +5 volts to pull the output high when the input is 0, and an NMOS transistor connected to ground to pull the output low when the input is 1. (The PMOS transistor needs to be larger because PMOS transistors don't function as well as NMOS transistors due to silicon physics.)
The polysilicon input line plays a key role: where it crosses the doped silicon, a transistor gate is formed. To make the standard cell more flexible, the input to the inverter can be connected on either the left or the right; in this case, the input is connected on the right and there is no connection on the left. The inverter's output can be taken from the polysilicon on the upper left or the right, but in this case, it is taken from the upper metal layer (not shown). The power, ground, and output lines are in the lower metal layer, which I have represented by the thin red, blue, and yellow lines. The black circles are connections between the metal layer and the underlying silicon.
This inverter appears dozens of times in the circuitry. However, I came across a few inverters that didn't make sense. The problem was that the inverter's output was connected to the output of a multiplexer. Since an inverter is either on or off, its value would clobber the output of the multiplexer.4 This didn't make any sense. I double- and triple-checked the wiring to make sure I hadn't messed up. After more investigation, I found another problem: the input to a "bad" inverter didn't make sense either. The input consisted of two signals shorted together, which doesn't work.
Finally, I realized what was going on. A "bad inverter" has the exact silicon layout of an inverter, but it wasn't an inverter: it was independent NMOS and PMOS transistors with separate inputs. Now it all made sense. With two inputs, the input signals were independent, not shorted together. And since the transistors were controlled separately, the NMOS transistor could pull the output low in some circumstances, the PMOS transistor could pull the output high in other circumstances, or both transistors could be off, allowing the multiplexer's output to be used undisturbed. In other words, the "inverter" was just two more cases for the multiplexer.
If you compare the "bad inverter" cell below with the previous cell, they look almost the same, but there are subtle differences. First, the gates of the two transistors are connected in the real inverter, but disconnected by a small gap in the transistor pair. I've indicated this gap in the photo above; it is hard to tell if the gap is real or just an imaging artifact, so I didn't spot it. The second difference is that the "fake" inverter has two input connections, one to each transistor, while the inverter has a single input connection. Unfortunately, I assumed that the two connections were just a trick to route the signal across the inverter without requiring an extra wire. In total, this cell was used 32 times as a real inverter and 9 times as independent transistors.
Standard cell logic and automatic place and route have a long history before the 386, back to the early 1970s, so this isn't an Intel invention.5 Nonetheless, the 386 team deserves the credit for deciding to use this technology at a time when it was a risky decision. They needed to develop custom software for their placing and routing needs, so this wasn't a trivial undertaking. This choice paid off and they completed the 386 ahead of schedule. The 386 ended up being a huge success for Intel, moving the x86 architecture to 32 bits and defining the dominant computer architecture for the rest of the 20th century.
If you're interested in standard cell logic, I also wrote about standard cell logic in an IBM chip. I plan to write more about the 386, so follow me on Mastodon, Bluesky, or RSS for updates. Thanks to Pat Gelsinger and Roxanne Koester for providing helpful papers.
For more on the 386 and other chips, follow me on Mastodon (@kenshirriff@oldbytes.space), Bluesky (@righto.com), or RSS. (I've given up on Twitter.) If you want to read more about the 386, I've written about the clock pin, prefetch queue, die versions, packaging, and I/O circuits.
The decision to use automatic place and route is described on page 13 of the Intel 386 Microprocessor Design and Development Oral History Panel, a very interesting document on the 386 with discussion from some of the people involved in its development. ↩
Multiplexers often take a binary control signal to select the desired input. For instance, an 8-to-1 multiplexer selects one of 8 inputs, so a 3-bit control signal can specify the desired input. The 386's multiplexers use a different approach with one control signal per input. One of the 8 control signals is activated to select the desired input. This approach is called a "one-hot encoding" since one control line is activated (hot) at a time. ↩
Some chips, such as the MOS Technology 6502 processor, are built with NMOS technology, without PMOS transistors. Multiplexers in the 6502 use a single NMOS transistor, rather than the two transistors in the CMOS switch. However, the performance of the switch is worse. ↩
One very common circuit in the 386 is a latch constructed from an inverter loop and a switch/multiplexer. The inverter's output and the switch's output are connected together. The trick, however, is that the inverter is constructed from special weak transistors. When the switch is disabled, the inverter's weak output is sufficient to drive the loop. But to write a value into the latch, the switch is enabled and its output overpowers the weak inverter.
The point of this is that there are circuits where an inverter and a multiplexer have their outputs connected. However, the inverter must be constructed with special weak transistors, which is not the situation that I'm discussing. ↩
I'll provide more history on standard cells in this footnote. RCA patented a bipolar standard cell in 1971, but this was a fixed arrangement of transistors and resistors, more of a gate array than a modern standard cell. Bell Labs researched standard cell layout techniques in the early 1970s, calling them Polycells, including a 1973 paper by Brian Kernighan. By 1979, A Guide to LSI Implementation discussed the standard cell approach and it was described as well-known in this patent application. Even so, Electronics called these design methods "futuristic" in 1980.
Standard cells became popular in the mid-1980s as faster computers and improved design software made it practical to produce semi-custom designs that used standard cells. Standard cells made it to the cover of Digital Design in August 1985, and the article inside described numerous vendors and products. Companies like Zymos and VLSI Technology (VTI) focused on standard cells. Traditional companies such as Texas Instruments, NCR, GE/RCA, Fairchild, Harris, ITT, and Thomson introduced lines of standard cell products in the mid-1980s. ↩
The New York Times recently introduced a new daily puzzle called Pips. You place a set of dominoes on a grid, satisfying various conditions. For instance, in the puzzle below, the pips (dots) in the purple squares must sum to 8, there must be fewer than 5 pips in the red square, and the pips in the three green squares must be equal. (It doesn't take much thought to solve this "easy" puzzle, but the "medium" and "hard" puzzles are more challenging.)
I was wondering about how to solve these puzzles with a computer. Recently, I saw an article on Hacker News—"Many hard LeetCode problems are easy constraint problems"—that described the benefits and flexibility of a system called a constraint solver. A constraint solver takes a set of constraints and finds solutions that satisfy the constraints: exactly what Pips requires.
I figured that solving Pips with a constraint solver would be a good way to learn more about these solvers, but I had several questions. Did constraint solvers require incomprehensible mathematics? How hard was it to express a problem? Would the solver quickly solve the problem, or would it get caught in an exponential search?
It turns out that using a constraint solver was straightforward; it took me under two hours from knowing nothing about constraint solvers to solving the problem. The solver found solutions in milliseconds (for the most part). However, there were a few bumps along the way. In this blog post, I'll discuss my experience with the MiniZinc1 constraint modeling system and show how it can solve Pips.
Writing a program for a constraint solver is very different from writing a regular program. Instead of telling the computer how to solve the problem, you tell it what you want: the conditions that must be satisfied. The solver then "magically" finds solutions that satisfy the problem.
To solve the problem, I created an array called pips that holds the number of domino pips at each position
in the grid.
Then, the three constraints for the above problem can be expressed as follows.
You can see how the constraints directly express the conditions in the puzzle.
constraint pips[1,1] + pips[2,1] == 8; constraint pips[2,3] < 5; constraint all_equal([pips[3,1], pips[3,2], pips[3,3]]);
Next, I needed to specify where dominoes could be placed for the puzzle.
To do this, I defined an array called grid that indicated the allowable positions: 1 indicates a valid
position and 0 indicates an invalid position. (If you compare with the puzzle at the top of the article,
you can see that the grid below matches its shape.)
grid = [| 1,1,0| 1,1,1| 1,1,1|];
I also defined the set of dominoes for the problem above, specifying the number of spots in each half:
spots = [|5,1| 1,4| 4,2| 1,3|];
So far, the constraints directly match the problem.
However, I needed to write some more code to specify how these pieces interact.
But
before I describe that code, I'll show a solution.
I wasn't sure what to expect: would the constraint solver give me a solution or would it spin
forever?
It turned out to find the unique solution in 109 milliseconds, printing out the
solution arrays.
The pips array shows the number of pips in each position, while the dominogrid array shows which
domino (1 through 4) is in each position.
pips = [| 4, 2, 0 | 4, 5, 3 | 1, 1, 1 |]; dominogrid = [| 3, 3, 0 | 2, 1, 4 | 2, 1, 4 |];
The text-based solution above is a bit ugly. But it is easy to create graphical output. MiniZinc provides a JavaScript API, so you can easily display solutions on a web page. I wrote a few lines of JavaScript to draw the solution, as shown below. (I just display the numbers since I was too lazy to draw the dots.) Solving this puzzle is not too impressive—it's an "easy" puzzle after all—but I'll show below that the solver can also handle considerably more difficult puzzles.
While the above code specifies a particular puzzle, a bit more code is required to define how dominoes and the grid interact. This code may appear strange because it is implemented as constraints, rather than the procedural operations in a normal program.
My main design decision was how to specify the locations of dominoes.
I considered assigning a grid position and orientation
to each domino, but it seemed inconvenient to deal with multiple orientations.
Instead, I decided to position each half of the domino independently, with an x and y coordinate in
the grid.2 I added a constraint that the two halves of each domino had to be in neighboring cells,
that is, either the X or Y coordinates had to differ by 1.
constraint forall(i in DOMINO) (abs(x[i, 1] - x[i, 2]) + abs(y[i, 1] - y[i, 2]) == 1);
It took a bit of thought to fill in the pips array with the number of spots on each domino.
In a normal programming language, one would loop over the dominoes and store the values into pips.
However, here it is done with a constraint so the solver makes sure the values are assigned.
Specifically, for each half-domino, the pips array entry at
the domino's x/y coordinate must equal the corresponding spots on the domino:
constraint forall(i in DOMINO, j in HALF) (pips[y[i,j], x[i, j]] == spots[i, j]);
I decided to add another array to keep track of which domino is in which position.
This array is useful to see the domino locations in the output, but it also
keeps dominoes from overlapping.
I used a constraint to put each domino's number (1, 2, 3, etc.) into the occupied position of dominogrid:
constraint forall(i in DOMINO, j in HALF) (dominogrid[y[i,j], x[i, j]] == i);
Next, how do we make sure that dominoes only go into positions allowed by grid?
I used a constraint that a square in dominogrid must be empty or the corresponding grid must allow a domino.3
This uses the "or" condition, which is expressed as \/, an unusual stylistic
choice. (Likewise, "and" is expressed as /\. These correspond to the logical symbols
∨ and ∧.)
constraint forall(i in 1..H, j in 1..W) (dominogrid[i, j] == 0 \/ grid[i, j] != 0);
Honestly, I was worried that I had too many arrays and the solver would end up in a rathole ensuring that the arrays were consistent. But I figured I'd try this brute-force approach and see if it worked. It turns out that it worked for the most part, so I didn't need to do anything more clever.
Finally, the program requires a few lines to define some constants and variables. The constants below define the number of dominoes and the size of the grid for a particular problem:
int: NDOMINO = 4; % Number of dominoes in the puzzle int: W = 3; % Width of the grid in this puzzle int: H = 3; % Height of the grid in this puzzle
Next, datatypes are defined to specify the allowable values.
This is very important for the solver; it is a "finite domain" solver, so limiting the size of
the domains reduces the size of the problem.
For this problem, the values are integers in a particular range, called a set:
set of int: DOMINO = 1..NDOMINO; % Dominoes are numbered 1 to NDOMINO set of int: HALF = 1..2; % The domino half is 1 or 2 set of int: xcoord = 1..W; % Coordinate into the grid set of int: ycoord = 1..H;
At last, I define the sizes and types of the various arrays that I use.
One very important syntax is var, which indicates variables that the solver must determine.
Note that the first two arrays, grid and spots do not have var since they are constant,
initialized to specify the problem.
array[1..H,1..W] of 0..1: grid; % The grid defining where dominoes can go array[DOMINO, HALF] of int: spots; % The number of spots on each half of each domino array[DOMINO, HALF] of var xcoord: x; % X coordinate of each domino half array[DOMINO, HALF] of var ycoord: y; % Y coordinate of each domino half array[1..H,1..W] of var 0..6: pips; % The number of pips (0 to 6) at each location. array[1..H,1..W] of var 0..NDOMINO: dominogrid; % The domino sequence number at each location
You can find all the code on GitHub.
One weird thing is that because the code is not procedural, the lines can be in any order.
You can use arrays or constants before you use them.
You can even move include statements to the end of the file if you want!
Overall, the solver was much easier to use than I expected. However, there were a few complications.
By changing a setting, the solver can find multiple solutions instead of stopping after the first. However, when I tried this, the solver generated thousands of meaningless solutions. A closer look showed that the problem was that the solver was putting arbitrary numbers into the "empty" cells, creating valid but pointlessly different solutions. It turns out that I didn't explicitly forbid this, so the sneaky constraint solver went ahead and generated tons of solutions that I didn't want. Adding another constraint fixed the problem. The moral is that even if you think your constraints are clear, solvers are very good at finding unwanted solutions that technically satisfy the constraints. 4
A second problem is that if you do something wrong, the solver simply says that the problem is unsatisfiable. Maybe there's a clever way of debugging, but I ended up removing constraints until the problem can be satisfied, and then see what I did wrong with that constraint. (For instance, I got the array indices backward at one point, making the problem insoluble.)
The most concerning issue is the unpredictability of the solver: maybe it will take milliseconds or maybe it will take hours. For instance, the Oct 5 hard Pips puzzle (below) caused the solver to take minutes for no apparent reason. However, the MiniZinc IDE supports different solver backends. I switched from the default Gecode solver to Chuffed, and it immediately found numerous solutions, 384 to be precise. (Sometimes the Pips puzzles sometimes have multiple solutions, which players find controversial.) I suspect that the multiple solutions messed up the Gecode solver somehow, perhaps because it couldn't narrow down a "good" branch in the search tree. For a benchmark of the different solvers, see the footnote.5
If you were writing a program to solve Pips from scratch, you'd probably have a loop to try assigning dominoes to positions. The problem is that the problem grows exponentially. If you have 16 dominoes, there are 16 choices for the first domino, 15 choices for the second, and so forth, so about 16! combinations in total, and that's ignoring orientations. You can think of this as a search tree: at the first step, you have 16 branches. For the next step, each branch has 15 sub-branches. Each sub-branch has 14 sub-sub-branches, and so forth.
An easy optimization is to check the constraints after each domino is added. For instance, as soon as the "less than 5" constraint is violated, you can backtrack and skip that entire section of the tree. In this way, only a subset of the tree needs to be searched; the number of branches will be large, but hopefully manageable.
A constraint solver works similarly, but in a more abstract way. The constraint solver assigns values to the variables, backtracking when a conflict is detected. Since the underlying problem is typically NP-complete, the solver uses heuristics to attempt to improve performance. For instance, variables can be assigned in different orders. The solver attempts to generate conflicts as soon as possible so large pieces of the search tree can be pruned sooner rather than later. (In the domino case, this corresponds to placing dominoes in places with the tightest constraints, rather than scattering them around the puzzle in "easy" spots.)
Another technique is constraint propagation. The idea is that you can derive new constraints and catch conflicts earlier. For instance, suppose you have a problem with the constraints "a equals c" and "b equals c". If you assign "a=1" and "b=2", you won't find a conflict until later, when you try to find a value for "c". But with constraint propagation, you can derive a new constraint "a equals b", and the problem will turn up immediately. (Solvers handle more complicated constraint propagation, such as inequalities.) The tradeoff is that generating new constraints takes time and makes the problem larger, so constraint propagation can make the solver slower. Thus, heuristics are used to decide when to apply constraint propagation.
Researchers are actively developing new algorithms, heuristics, and optimizations6 such as backtracking more aggressively (called "backjumping"), keeping track of failing variable assignments (called "nogoods"), and leveraging Boolean SAT (satisfiability) solvers. Solvers compete in annual challenges to test these techniques against each other. The nice thing about a constraint solver is that you don't need to know anything about these techniques; they are applied automatically.
I hope this has convinced you that constraint solvers are interesting, not too scary, and can solve real problems with little effort. Even as a beginner, I was able to get started with MiniZinc quickly. (I read half the tutorial and then jumped into programming.)
One reason to look at constraint solvers is that they are a completely different programming paradigm. Using a constraint solver is like programming on a higher level, not worrying about how the problem gets solved or what algorithm gets used. Moreover, analyzing a problem in terms of constraints is a different way of thinking about algorithms. Some of the time it's frustrating when you can't use familiar constructs such as loops and assignments, but it expands your horizons.
Finally, writing code to solve Pips is more fun than solving the problems by hand, at least in my opinion, so give it a try!
For more, follow me on Bluesky (@righto.com), Mastodon (@kenshirriff@oldbytes.space), RSS, or subscribe here.
all_equal and alldifferent constraint functions.I started by downloading the MiniZinc IDE and reading the MiniZinc tutorial. The MiniZinc IDE is straightforward, with an editor window at the top and an output window at the bottom. Clicking the "Run" button causes it to generate a solution.
It might be cleaner to combine the X and Y coordinates into a single Point type, using a MiniZinc record type. ↩
I later decided that it made more sense to enforce that dominogrid is empty if and only if
grid is 0 at that point, although it doesn't affect the solution.
This constraint uses the "if and only if" operator <->.
constraint forall(i in 1..H, j in 1..W) (dominogrid[i, j] == 0 <-> grid[i, j] == 0);↩
To prevent the solver from putting arbitrary numbers in the unused positions of pips, I added a
constraint to force these values to be zero:
constraint forall(i in 1..H, j in 1..W) (grid[i, j] == 0 -> pips[i, j] == 0);
Generating multiple solutions had a second issue, which I expected: A symmetric domino can be placed in two redundant ways. For instance, a double-six domino can be flipped to produce a solution that is technically different but looks the same. I fixed this by adding constraints for each symmetric domino to allow only one of the two redundant positions. The constraint below forces a preferred orientation for symmetric dominoes.
constraint forall(i in DOMINO) (spots[i,1] != spots[i,2] \/ x[i,1] > x[i,2] \/ (x[i,1] == x[i,2] /\ y[i,1] > y[i,2]));
To enable multiple solutions in MiniZinc, the setting is under Show Configuration Editor > User Defined Behavior >
Satisfaction Problems or the --all flag from the command line. ↩
MiniZinc has five solvers that can solve this sort of integer problem: Chuffed, OR Tools CP-SAT, Gecode, HiGHS, and Coin-OR BC. I measured the performance of the five solvers against 20 different Pips puzzles. Most of the solvers found solutions in under a second, most of the time, but there is a lot of variation.
Overall, Chuffed had the best performance on the puzzles that I tested, taking well under a second. Google's OR-Tools won all the categories in the 2025 MiniZinc challenge, but it was considerably slower than Chuffed for my Pips programs. The default Gecode solver performed very well most of the time, but it did terribly on a few problems, taking over 15 minutes. HiGHs was slower in general, taking a few minutes on the hardest problems, but it didn't fail as badly as Gecode. (Curiously, Gecode and HiGHS sometimes found different problems to be difficult.) Finally, Coin-OR BC was uniformly bad; at best it took a few seconds, but one puzzle took almost two hours and others weren't solved before I gave up after two hours. (I left Coin-OR BC off the graph because it messed up the scale.)
Don't treat these results too seriously because different solvers are optimized for different purposes. (In particular, Coin-OR BC is designed for linear problems.) But the results demonstrate the unpredictability of solvers: maybe you get a solution in a second and maybe you get a solution in hours. ↩
If you want to read more about solvers, Constraint Satisfaction Problems is an overview presentation. The Gecode algorithms are described in a nice technical report: Constraint Programming Algorithms used in Gecode. Chuffed is more complicated: "Chuffed is a state of the art lazy clause solver designed from the ground up with lazy clause generation in mind. Lazy clause generation is a hybrid approach to constraint solving that combines features of finite domain propagation and Boolean satisfiability." The Chuffed paper Lazy clause generation reengineered and slides are more of a challenge. ↩
The noted Diné (Navajo) weaver Marilou Schultz recently completed an intricate weaving composed of thick white lines on a black background, punctuated with reddish-orange diamonds. Although this striking rug may appear abstract, it shows the internal circuitry of a tiny silicon chip known as the 555 timer. This chip has hundreds of applications in everything from a sound generator to a windshield wiper controller. At one point, the 555 was the world's best-selling integrated circuit with billions sold. But how did the chip get turned into a rug?
The 555 chip is constructed from a tiny flake of silicon with a layer of metallic wiring on top. In the rug, this wiring is visible as the thick white lines, while the silicon forms the black background. One conspicuous feature of the rug is the reddish-orange diamonds around the perimeter. These correspond to the connections between the silicon chip and its eight pins. Tiny golden bond wires—thinner than a human hair—are attached to the square bond pads to provide these connections. The circuitry of the 555 chip contains 25 transistors, silicon devices that can switch on and off. The rug is dominated by three large transistors, the filled squares with a 王 pattern inside, while the remaining transistors are represented by small dots.
The weaving was inspired by a photo of the 555 timer die taken by Antoine Bercovici (Siliconinsider); I suggested this photo to Schultz as a possible subject for a rug. The diagram below compares the weaving (left) with the die photo (right). As you can see, the weaving closely follows the actual chip, but there are a few artistic differences. For instance, two of the bond pads have been removed, the circuitry at the top has been simplified, and the part number at the bottom has been removed.
Antoine took the die photo with a dark field microscope, a special type of microscope that produces an image on a black background. This image emphasizes the metal layer on the top of the die. In comparison, a standard bright-field microscope produced the image below. When a chip is manufactured, regions of silicon are "doped" with impurities to create transistors and resistors. These regions are visible in the image below as subtle changes in the color of the silicon.
In the weaving, the chip's design appears almost monumental, making it easy to forget that the actual chip is microscopic. For the photo below, I obtained a version of the chip packaged in a metal can, rather than the typical rectangle of black plastic. Cutting the top off the metal can reveals the tiny chip inside, with eight gold bond wires connecting the die to the pins of the package. If you zoom in on the photo, you may recognize the three large transistors that dominate the rug.
The artist, Marilou Schultz, has been creating chip rugs since 1994, when Intel commissioned a rug based on the Pentium as a gift to AISES (American Indian Science & Engineering Society). Although Schultz learned weaving as a child, the Pentium rug was a challenge due to its complex pattern and lack of symmetry; a day's work might add just an inch to the rug. This dramatic weaving was created with wool from the long-horned Navajo-Churro sheep, colored with traditional plant dyes.
For the 555 timer weaving, Schultz experimented with different materials. Silver and gold metallic threads represent the aluminum and copper in the chip. The artist explains that "it took a lot more time to incorporate the metallic threads," but it was worth the effort because "it is spectacular to see the rug with the metallics in the dark with a little light hitting it." Aniline dyes provided the black and lavender colors. Although natural logwood dye produces a beautiful purple, it fades over time, so Schultz used an aniline dye instead. The lavender colors are dedicated to the weaver's mother, who passed away in February; purple was her favorite color.
How does the 555 chip produce a particular time delay? You add external components—resistors and a capacitor—to select the time. The capacitor is filled (charged) at a speed controlled by the resistor. When the capacitor get "full", the 555 chip switches operation and starts emptying (discharging) the capacitor. It's like filling a sink: if you have a large sink (capacitor) and a trickle of water (large resistor), the sink fills slowly. But if you have a small sink (capacitor) and a lot of water (small resistor), the sink fills quickly. By using different resistors and capacitors, the 555 timer can provide time intervals from microseconds to hours.
I've constructed an interactive chip browser that shows how the regions of the rug correspond to specific electronic components in the physical chip. Click on any part of the rug to learn the function of the corresponding component in the chip.
For instance, two of the large square transistors turn the chip's output on or off, while the third large transistor discharges the capacitor when it is full. (To be precise, the capacitor goes between 1/3 full and 2/3 full to avoid issues near "empty" and "full".) The chip has circuits called comparators that detect when the capacitor's voltage reaches 1/3 or 2/3, switching between emptying and filling at those points. If you want more technical details about the 555 chip, see my previous articles: an early 555 chip, a 555 timer similar to the rug, and a more modern CMOS version of the 555.
The similarities between Navajo weavings and the patterns in integrated circuits have long been recognized. Marilou Schultz's weavings of integrated circuits make these visual metaphors into concrete works of art. This connection is not just metaphorical, however; in the 1960s, the semiconductor company Fairchild employed numerous Navajo workers to assemble chips in Shiprock, New Mexico. I wrote about this complicated history in The Pentium as a Navajo Weaving.
This work is being shown at SITE Santa Fe's Once Within a Time exhibition (running until January 2026). I haven't seen the exhibition in person, so let me know if you visit it. For more about Marilou Schultz's art, see The Diné Weaver Who Turns Microchips Into Art, or A Conversation with Marilou Schultz on YouTube.
Many thanks to Marilou Schultz for discussing her art with me. Thanks to First American Art Magazine for providing the photo of her 555 rug. Follow me on Mastodon (@kenshirriff@oldbytes.space), Bluesky (@righto.com), or RSS for updates.
I was reading the latest issue of the journal Science, and a paper mentioned the compound Cr2Gr2Te6. For a moment, I thought my knowledge of the periodic table was slipping, since I couldn't remember the element Gr. It turns out that Gr was supposed to be Ge, germanium, but that raises two issues. First, shouldn't the peer reviewers and proofreaders at a top journal catch this error? But more curiously, it appears that this formula is a mistake that has been copied around several times.
The Science paper [1] states, "Intrinsic ferromagnetism in these materials was discovered in Cr2Gr2Te6 and CrI3 down to the bilayer and monolayer thickness limit in 2017." I checked the referenced paper [2] and verified that the correct compound is Cr2Ge2Te6, with Ge for germanium.
But in the process, I found more publications that specifically mention the 2017 discovery of intrinsic ferromagnetism in both Cr2Gr2Te6 and CrI3. A 2021 paper in Nanoscale [3] says, "Since the discovery of intrinsic ferromagnetism in atomically thin Cr2Gr2Te6 and CrI3 in 2017, research on two-dimensional (2D) magnetic materials has become a highlighted topic." Then, a 2023 book chaper [4] opens with the abstract: "Since the discovery of intrinsic long-range magnetic order in two-dimensional (2D) layered magnets, e.g., Cr2Gr2Te6 and CrI3 in 2017, [...]"
This illustrates how easy it is for a random phrase to get copied around with nobody checking it. (Earlier, I found a bogus computer definition that has persisted for over 50 years.) To be sure, these could all be independent typos—it's an easy typo to make since Ge and Gr are neighbors on the keyboard and Cr2Gr2 scans better than Cr2Ge2. A few other papers [5, 6, 7] have the same typo, but in different contexts. My bigger concern is that once AI picks up the erroneous formula, it will propagate as misinformation forever. I hope that by calling out this error, I can bring an end to it. In any case, if anyone ends up here after a web search, I can at least confirm that there isn't a new element Gr and the real compound is Cr2Ge2Te6, chromium germanium telluride.
[1] He, B. et al. (2025) ‘Strain-coupled, crystalline polymer-inorganic interfaces for efficient magnetoelectric sensing’, Science, 389(6760), pp 623-631. (link)
[2] Gong, C. et al. (2017) ‘Discovery of intrinsic ferromagnetism in two-dimensional van der Waals crystals’, Nature, 546(7657), pp. 265–269. (link)
[3] Zhang, S. et al. (2021) ‘Two-dimensional magnetic materials: structures, properties and external controls’, Nanoscale, 13(3), pp. 1398–1424. (link)
[4] Yin, T. (2024) ‘Novel Light-Matter Interactions in 2D Magnets’, in D. Ranjan Sahu (ed.) Modern Permanent Magnets - Fundamentals and Applications. (link)
[5] Zhao, B. et al. (2023) ‘Strong perpendicular anisotropic ferromagnet Fe3GeTe2/graphene van der Waals heterostructure’, Journal of Physics D: Applied Physics, 56(9) 094001. (link)
[6] Ren, H. and Lan, M. (2023) ‘Progress and Prospects in Metallic FexGeTe2 (3≤x≤7) Ferromagnets’, Molecules, 28(21), p. 7244. (link)
[7] Hu, S. et al. (2019) 'Anomalous Hall effect in Cr2Gr2Te6/Pt hybride structure', Taiwan-Japan Joint Workshop on Condensed Matter Physics for Young Researchers, Saga, Japan. (link)
I've been reverse-engineering the Intel 386 processor (from 1985), and I've come across some interesting circuits for the chip's input/output (I/O) pins. Since these pins communicate with the outside world, they face special dangers: static electricity and latchup can destroy the chip, while metastability can cause serious malfunctions. These I/O circuits are completely different from the logic circuits in the 386, and I've come across a previously-undescribed flip-flop circuit, so I'm venturing into uncharted territory. In this article, I take a close look at how the I/O circuitry protects the 386 from the "dragons" that can destroy it.
The photo above shows the die of the 386 under a microscope. The dark, complex patterns arranged in rectangular regions arise from the two layers of metal that connect the circuits on the 386 chip. Not visible are the transistors, formed from silicon and polysilicon and hidden beneath the metal. Around the perimeter of this fingernail-sized silicon die, 141 square bond pads provide the connections between the chip and the outside world; tiny gold bond wires connect the bond pads to the package. Next to each I/O pad, specialized circuitry provides the electrical interface between the chip and the external components while protecting the chip. I've zoomed in on three groups of these bond pads along with the associated I/O circuits. The circuits at the top (for data pins) and the left (for address pins) are completely different from the control pin circuits at the bottom, showing how the circuitry varies with the pin's function.
The first dragon that threatens the 386 is static electricity, able to burn a hole in the chip. MOS transistors are constructed with a thin insulating oxide layer underneath the transistor's gate. In the 386, this fragile, glass-like oxide layer is just 250 nm thick, the thickness of a virus. Static electricity, even a small amount, can blow a hole through this oxide layer and destroy the chip. If you've ever walked across a carpet and felt a spark when you touch a doorknob, you've generated at least 3000 volts of chip-destroying static electricity. Intel recommends an anti-static mat and a grounding wrist strap when installing a processor to avoid the danger of static electricity, also known as Electrostatic Discharge or ESD.1
To reduce the risk of ESD damage, chips have protection diodes and other components in their I/O circuitry. The schematic below shows the circuit for a typical 386 input. The goal is to prevent static discharge from reaching the inverter, where it could destroy the inverter's transistors. The diodes next to the pad provide the first layer of protection; they redirect excess voltage to the +5 rail or ground. Next, the resistor reduces the current that can reach the inverter. The third diode provides a final layer of protection. (One unusual feature of this input—unrelated to ESD—is that the input has a pull-up, which is implemented with a transistor that acts like a 20kΩ resistor.2)
BS16# pad circuit. The BS16# signal indicates to the 386 if the external bus is 16 bits or 32 bits.The image below shows how this circuit appears on the die. For this photo, I dissolved the metal layers with acids, stripping the die down to the silicon to make the transistors visible. The diodes and pull-up resistor are implemented with transistors.3 Large grids of transistors form the pad-side diodes, while the third diode is above. The current-limiting protection resistor is implemented with polysilicon, which provides higher resistance than metal wiring. The capacitor is implemented with a plate of polysilicon over silicon, separated by a thin oxide layer. As you can see, the protection circuitry occupies much more area than the inverters that process the signal.
The transistors in the 386 are created by doping silicon with impurities to change its properties, creating regions of "N-type" and "P-type" silicon. The 386 chip, like most processors, is built from CMOS technology, so it uses two types of transistors: NMOS and PMOS. The 386 starts from a wafer of N-type silicon and PMOS transistors are formed by doping tiny regions to form P-type silicon embedded in the underlying N-type silicon. NMOS transistors are the opposite, with N-type silicon embedded in P-type silicon. To hold the NMOS transistors, "wells" of P-type silicon are formed, as shown in the cross-section diagram below. Thus, the 386 chip contains complex patterns of P-type and N-type silicon that form its 285,000 transistors.
But something dangerous lurks below the surface, the fire-breathing dragon of latchup waiting to burn up the chip. The problem is that these regions of N-type and P-type silicon form unwanted, "parasitic" transistors underneath the desired transistors. In normal circumstances, these parasitic NPN and PNP transistors are inactive and can be ignored. But if a current flows beneath the surface, through the silicon substrate, it can turn on a parasitic transistor and awaken the dreaded latchup.4 The parasitic transistors form a feedback loop, so if one transistor starts to turn on, it turns on the other transistor, and so forth, until both transistors are fully on, a state called latchup.5 Moreover, the feedback loop will maintain latchup until the chip's power is removed.6 During latchup, the chip's power and ground are shorted through the parasitic transistors, causing high current flow that can destroy the chip by overheating it or even melting bond wires.
Latchup can be triggered in many ways, from power supply overvoltage to radiation, but a chip's I/O pins are the primary risk because signals from the outside world are unpredictable. For instance, suppose a floppy drive is connected to the 386 and the drive sends a signal with a voltage higher than the 386's 5-volt supply. (This could happen due to a voltage surge in the drive, reflection in a signal line, or even connecting a cable.) Current will flow through the 386's protection diodes, the diodes that were described in the previous section.7 If this current flows through the chip's silicon substrate, it can trigger latchup and destroy the processor.
Because of this danger, the 386's I/O pads are designed to prevent latchup. One solution is to block the unwanted currents through the substrate, essentially putting fences around the transistors to keep malicious currents from escaping into the substrate. In the 386, this fence consists of "guard rings" around the I/O transistors and diodes. These rings prevent latchup by blocking unwanted current flow and safely redirecting it to power or ground.
The diagram above shows the double guard rings for a typical I/O pad.8 Separate guard rings protect the NMOS transistors and the PMOS transistors. The NMOS transistors have an inner guard ring of P-type silicon connected to ground (blue) and an outer guard ring of N-type silicon connected to +5 (red). The rings are reversed for the PMOS transistors. The guard rings take up significant space on the die, but this space isn't wasted since the rings protect the chip from latchup.
The final dragon is metastability: it (probably) won't destroy the chip, but it can cause serious malfunctions.9 Metastability is a peculiar problem where a digital signal can take an unbounded amount of time to settle into a zero or a one. In other words, the circuit temporarily refuses to act digitally and shows its underlying analog nature.10 Metastability was controversial in the 1960s and the 1970s, with many electrical engineers not believing it existed or considering it irrelevant. Nowadays, metastability is well understood, with special circuits to prevent it, but metastability can never be completely eliminated.
In a processor, everything is synchronized to its clock. While a modern processor has a clock speed of several gigahertz, the 386's clock ran at 12 to 33 megahertz. Inside the processor, signals are carefully organized to change according to the clock—that's why your computer runs faster with a higher clock speed. The problem is that external signals may be independent of the CPU's clock. For instance, a disk drive could send an interrupt to the computer when data is ready, which depends on the timing of the spinning disk. If this interrupt arrives at just the wrong time, it can trigger metastability.
In more detail, processors use flip-flops to hold signals under the control of the clock. An "edge-triggered" flip-flop grabs its input at the moment the clock goes high (the "rising edge") and holds this value until the next clock cycle. Everything is fine if the value is stable when the clock changes: if the input signal switches from low to high before the clock edge, the flip-flop will hold this high value. And if the input signal switches from low to high after the clock edge, the flip-flop will hold the low value, since the input was low at the clock edge. But what happens if the input changes from low to high at the exact time that the clock switches? Usually, the flip-flop will pick either low or high. But very rarely, maybe a few times out of a billion, the flip-flop will hesitate in between, neither low nor high. The flip-flop may take a few nanoseconds before it "decides" on a low or high value, and the value will be intermediate until then.
The photo above illustrates a metastable signal, spending an unpredictable time between zero and one before settling on a value. The situation is similar to a ball balanced on top of a hill, a point of unstable equilibrium.11 The smallest perturbation will knock the ball down one of the two stable positions at the bottom of the hill, but you don't know which way it will go or how long it will take.
Metastability is serious because if a digital signal has a value that is neither 0 nor 1 then downstream circuitry may get confused. For instance, if part of the processor thinks that it received an interrupt and other parts of the processor think that no interrupt happened, chaos will reign as the processor takes contradictory actions. Moreover, waiting a few nanoseconds isn't a cure because the duration of metastability can be arbitrarily long. Waiting helps, since the chance of metastability decreases exponentially with time, but there is no guarantee.12
The obvious solution is to never change an input exactly when the clock changes. The processor is designed so that internal signals are stable when the clock changes, avoiding metastability. Specifically, the designer of a flip-flop specifies the setup time—how long the signal must be stable before the clock edge—and the hold time—how long the signal must be stable after the clock edge. As long as the input satisfies these conditions, typically a few picoseconds long, the flip-flop will function without metastability.
Unfortunately, the setup and hold times can't be guaranteed when the processor receives an external signal that isn't synchronized to its clock, known as an asynchronous signal. For instance, a processor receives interrupt signals when an I/O device has data, but the timing is unpredictable because it depends on mechanical factors such as a keypress or a spinning floppy disk. Most of the time, everything will work fine, but what about the one-in-a-billion case where the timing of the signal is unlucky? (Since modern processors run at multi-gigahertz, one-in-a-billion events are not rare; they can happen multiple times per second.)
One solution is a circuit called a synchronizer that takes an asynchronous signal and synchronizes it to the clock. A synchronizer can be implemented with two flip-flops in series: even if the first flip-flop has a metastable output, chances are that it will resolve to 0 or 1 before the second flip-flop stores the value. Each flip-flop provides an exponential reduction in the chance of metastability, so using two flip-flops drastically reduces the risk. In other words, the circuit will still fail occasionally, but if the mean time between failures (MTBF) is long enough (say, decades instead of seconds), then the risk is acceptable.
The schematic above shows how the 386 uses two flip-flops to minimize metastability. The first flip-flop is a special flip-flop that is based on a sense amplifier. It is much more complicated than a regular flip-flop, but it responds faster, reducing the chance of metastability. It is built from two of the sense-amplifier latches below, which I haven't seen described anywhere. In a DRAM memory chip, a sense amplifier takes a weak signal from a memory cell and rapidly amplifies it into a solid 0 or 1. In this flip-flop, the sense amplifier takes a potentially ambiguous signal and rapidly amplifies it into a 0 or 1. By amplifying the signal quickly, the flip-flop reduces metastability. (See the footnote for details.14)
The die photo below shows how this circuitry looks on the die. Each flip-flop is built from two latches; note that the sense-amp latches are larger than the standard latches. As before, the pad has protection diodes inside guard rings. For some reason, however, these diodes have a different structure from the transistor-based diodes described earlier. The 386 has five inputs that use this circuitry to protect against metastability.13 These inputs are all located together at the bottom of the die—it probably makes the layout more compact when neighboring pad circuits are all the same size.
In summary, the 386's I/O circuits are interesting because they are completely different from the chip's regular logic circuitry. In these circuits, the border between digital and analog breaks down; these circuits handle binary signals, but analog issues dominate the design. Moreover, hidden parasitic transistors play key roles; what you don't see can be more important than what you see. These circuits defend against three dangerous "dragons": static electricity, latchup, and metastability. Intel succeeded in warding off these dragons and the 386 was a success.
For more on the 386 and other chips, follow me on Mastodon (@kenshirriff@oldbytes.space), Bluesky (@righto.com), or RSS. (I've given up on Twitter.) If you want to read more about 386 input circuits, I wrote about the clock pin here
Anti-static precautions are specified in Intel's processor installation instructions. Also see Intel's Electrostatic Discharge and Electrical Overstress Guide. I couldn't find ESD ratings for the 386, but a modern Intel chip is tested to withstand 500 volts or 2000 volts, depending on the test procedure. ↩
The BS16# pin is slightly unusual because it has an internal pull-up resistor. If you look at the datasheet (9.2.3 and Table 9-3 footnotes), a few input pins (ERROR#, BUSY#, and BS16#) have internal pull-up resistors of 20 kΩ, while the PEREQ input pin has an internal pull-down resistor of 20 kΩ. ↩
The protection diode is probably a grounded-gate NMOS (ggNMOS), an NMOS transistor with the gate, source, and body (but not the drain) tied to ground. This forms a parasitic NPN transistor under the MOSFET that dissipates the ESD. (I think that the PMOS protection is the same, except the gate is pulled high, not grounded.) For output pins, the output driver MOSFETs have parasitic transistors that make the output driver "self-protected". One consequence is that the input pads and the output pads look similar (both have large MOS transistors), unlike other chips where the presence of large transistors indicates an output. (Even so, 386 outputs and inputs can be distinguished because outputs have large inverters inside the guard rings to drive the MOSFETs, while inputs do not.) Also see Practical ESD Protection Design. ↩
The 386 uses P-wells in an N-doped substrate. The substrate is heavily doped with antimony, with a lightly doped N epitaxial layer on top. This doping helped provide immunity to latchup. (See "High performance technology, circuits and packaging for the 80386", ICCD 1986.) For the most part, modern chips use the opposite: N-wells with a P-doped substrate. Why the substrate change?
In the earlier days of CMOS, P-well was standard due to the available doping technology, see N-well and P-well performance comparison. During the 1980s, there was controversy over which was better: P-well or N-well: "It is commonly agreed that P-well technology has a proven reliability record, reduced alpha-particle sensitivity, closer matched p- and n- channel devices, and high gain NPN structures. N-well proponents acknowledge better compatibility and performance with NMOS processing and designs, good substrate quality, availability, and cost, lower junction capacitance, and reduced body effects." (See Design of a CMOS Standard Cell Library.)
As wafer sizes increased in the 1990s, technology shifted to P-doped substrates because it is difficult to make large N-doped wafers due to the characteristics of the dopants (link). Some chips optimize transistor characteristics by using both types of wells, called a twin-well process. For instance, the Pentium used P-doped wafers and implanted both N and P wells. (See Intel's 0.25 micron, 2.0 volts logic process technology.) ↩
You can also view the parasitic transistors as forming an SCR (Silicon Controlled Rectifier), a four-layer semiconductor device. SCRs were popular in the 1970s because they could handle higher currents and voltages than transistors. But as high-power transistors were developed, SCRs fell out of favor. In particular, once an SCR is turned on, it stays on until power is removed or reversed; this makes SCRs harder to use than transistors. (This is the same characteristic that makes latchup so dangerous.) ↩
Satellites and nuclear missiles have a high risk of latchup due to radiation. Since radiation-induced latchup cannot always be prevented, one technique for dealing with latchup is to detect the excessive current from latchup and then power-cycle the chip. For instance, you can buy a radiation-hardened current limiter chip that will detect excessive current due to latchup and temporarily remove power; this chip sells for the remarkable price of $1780.
For more on latchup, see the Texas Instruments Latch-Up white paper, as well as Latch-Up, ESD, and Other Phenomena. ↩
The 80386 Hardware Reference Manual discusses how a computer designer can prevent latchup in the 386. The designer is assured that Intel's "CHMOS III" process prevents latchup under normal operating conditions. However, exceeding the voltage limits on I/O pins can cause current surges and latchup. Intel provides three guidelines: observe the maximum ratings for input voltages, never apply power to a 386 pin before the chip is powered up, and terminate I/O signals properly to avoid overshoot and undershoot. ↩
The circuit for the WR# pin is similar to many other output pins.
The basic idea is that a large PMOS transistor pulls the output high, while a large NMOS transistor pulls the
output low.
If the enable input is low, both transistors are turned off and the output floats. (This allows other devices
to take over the bus in the HOLD state.)
The inverters that control the drive transistors have an unusual layout. These inverters are inside the guard rings, meaning that the inverters are split apart, with the NMOS transistors in one ring and PMOS transistors in the other. The extra wiring adds capacitance to the output which probably makes the inverters slightly slower.
These inverters have a special design: one inverter is faster to go high than to go low, while the other inverter is the opposite. The motivation is that if both drive transistors are on at the same time, a large current will flow through the transistors from power to ground, producing an unwanted current spike (and potentially latchup). To avoid this, the inverters are designed to turn one drive transistor off faster than turning the other one on. Specifically, the high-side inverter has an extra transistor to quickly pull its output high, while the low-side inverter has an extra transistor to pull the output low. Moreover, the inverter's extra transistor is connected directly to the drive transistors, while the inverter's main output connects through a longer polysilicon path with more resistance, providing an RC delay. I found this layout very puzzling until I realized that the designers were carefully controlling the turn-on and turn-off speeds of these inverters. ↩
In Metastability and Synchronizers: A Tutorial, there's a story of a spacecraft power supply being destroyed by metastability. Supposedly, metastability caused the logic to turn on too many units, overloading and destroying the power supply. I suspect that this is a fictional cautionary tale, rather than an actual incident.
For more on metastability, see this presentation and this writeup by Tom Chaney, one of the early investigators of metastability. ↩
One of Vonada's Engineering Maxims is "Digital circuits are made from analog parts." Another maxim is "Synchronizing circuits may take forever to make a decision." These maxims and a dozen others are from Don Vonada in DEC's 1978 book Computer Engineering. ↩
Curiously, the definition of metastability in electronics doesn't match the definition in physics and chemistry. In electronics, a metastable state is an unstable equilibrium. In physics and chemistry, however, a metastable state is a stable state, just not the most stable ground state, so a moderate perturbation will knock it from the metastable state to the ground state. (In the hill analogy, it's as if the ball is caught in a small basin partway down the hill.) ↩
In case you're wondering what's going on with metastability at the circuit level, I'll give a brief explanation. A typical flip-flop is based on a latch circuit like the one below, which consists of two inverters and an electronic switch controlled by the clock. When the clock goes high, the inverters are configured into a loop, latching the prior input value. If the input was high, the output from the first inverter is low and the output from the second inverter is high. The loop feeds this output back into the first inverter, so the circuit is stable. Likewise, the circuit can be stable with a low input.
But what happens if the clock flips the switch as the input is changing, so the input to the first inverter is somewhere between zero and one? We need to consider that an inverter is really an analog device, not a binary device. You can describe it by a "voltage transfer curve" (purple line) that specifies the output voltage for a particular input voltage. For example, if you put in a low input, you get a high output, and vice versa. But there is an equilibrium point where the output voltage is the same as the input voltage. This is where metastability happens.
Suppose the input voltage to the inverter is the equilibrium voltage. It's not going to be precisely the equilibrium voltage (because of noise if nothing else), so suppose, for example, that it is 1µV above equilibrium. Note that the transfer curve is very steep around equilibrium, say a slope of 100, so it will greatly amplify the signal away from equilibrium. Thus, if the input is 1µV above equilibrium, the output will be 100µV below equilibrium. Then the next inverter will amplify again, sending a signal 10mV above equilibrium back to the first inverter. The distance will be amplified again, now 1000mV below equilibrium. At this point, you're on the flat part of the curve, so the second inverter will output +5V and the first inverter will output 0V, and the circuit is now stable.
The point of this is that the equilibrium voltage is an unstable equilibrium, so the circuit will eventually settle into the +5V or 0V states. But it may take an arbitrary number of loops through the inverters, depending on how close the starting point was to equilibrium. (The signal is continuous, so referring to "loops" is a simplification.) Also note that the distance from equilibrium is amplified exponentially with time. This is why the chance of metastability decreases exponentially with time. ↩
Looking at the die shows that the pins with metastability protection are INTR, NMI, PEREQ, ERROR#,
and BUSY#.
The 80386 Hardware Reference Manual lists these same five pins as asynchronous—I like it when I spot something
unusual on the die and then discover that it matches an obscure statement in the documentation.
The interrupt pins INTR and NMI are asynchronous because they come from external sources that may not
be using the 386's clock. But what about
PEREQ, ERROR#, and BUSY#?
These pins are part of the interface with an external math coprocessor (the 287 or 387 chip).
In most cases, the coprocessor uses the 386's clock. However, the 387 supported a little-used asynchronous mode
where the processor and the coprocessor could run at different speeds. ↩
The 386's metastability flip-flop is constructed with an unusual circuit. It has two latch stages (which is normal), but instead of using two inverters in a loop, it uses a sense-amplifier circuit. The idea of the sense amplifier is that it takes a differential input. When the clock enables the sense amplifier, it drives the higher input high and the lower input low (the inputs are also the outputs). (Sense amplifiers are used in dynamic RAM chips to amplify the tiny signals from a RAM cell to form a 0 or 1. At the same time, the amplifier refreshes the DRAM cell by generating full voltages.) Note that the sense amplifier's inputs also act as outputs; inputs during clock phase 1 and outputs during phase 2.
The schematic shows one of the latch stages; the complete flip-flop has a second stage, identical except that the clock phases are switched. This latch is much more complex than the typical 386 latch; 14 transistors versus 6 or 8. The sense amplifier is similar to two inverters in a loop, except they share a limited power current and a limited ground current. As one inverter starts to go high, it "steals" the supply current from the other. Meanwhile, the other inverter "steals" the ground current. Thus, a small difference in inputs is amplified, just as in a differential amplifier. Thus, by combining the amplification of a differential amplifier with the amplification of the inverter loop, this circuit reaches its final state faster than a regular inverter loop.
In more detail, during the first clock phase, the two inverters at the top generate the inverted and non-inverted signals. (In a metastable situation, these will be close to the midpoint, not binary.) During the second clock phase, the sense amplifier is activated. You can think of it as a differential amplifier with cross-coupling. If one input is slightly higher than the other, the amplifier pulls that input higher and the input lower, amplifying the difference. (The point is to quickly make the difference large enough to resolve the metastability.)
I couldn't find any latches like this in the literature. Comparative Analysis and Study of Metastability on High-Performance Flip-Flops describes eleven high-performance flip-flops. It includes two flip-flops that are based on sense amplifiers, but their circuits are very different from the 386 circuit. Perhaps the 386 circuit is an Intel design that was never publicized. In any case, let me know if this circuit has an official name. ↩
Intel released the 386 processor in 1985, the first 32-bit chip in the x86 line. This chip was packaged in a ceramic square with 132 gold-plated pins protruding from the underside, fitting into a socket on the motherboard. While this package may seem boring, a lot more is going on inside it than you might expect. Lumafield performed a 3-D CT scan of the chip for me, revealing six layers of complex wiring hidden inside the ceramic package. Moreover, the chip has nearly invisible metal wires connected to the sides of the package, the spikes below. The scan also revealed that the 386 has two separate power and ground networks: one for I/O and one for the CPU's logic.
The package, below, provides no hint of the complex wiring embedded inside the ceramic. The silicon die is normally not visible, but I removed the square metal lid that covers it.1 As a result, you can also see the two tiers of gold contacts that surround the silicon die.
Intel selected the 132-pin ceramic package to meet the requirements of a high pin count, good thermal characteristics, and low-noise power to the die.2: However, standard packages didn't provide sufficient power, so Intel designed a custom package with "single-row double shelf bonding to two signal layers and four power and ground planes." In other words, the die's bond wires are connected to the two shelves (or tiers) of pads surrounding the die. Internally, the package is like a 6-layer printed-circuit board made from ceramic.
The photo below shows the two tiers of pads with tiny gold bond wires attached: I measured the bond wires at 35 µm in diameter, thinner than a typical human hair. Some pads have up to five wires attached to support more current for the power and ground pads. You can consider the package to be a hierarchical interface from the tiny circuits on the die to the much larger features of the computer's motherboard. Specifically, the die has a feature size of 1 µm, while the metal wiring on top of the die has 6 µm spacing. The chip's wiring connects to the chip's bond pads, which have 0.01" spacing (.25 mm). The bond wires connect to the package's pads, which have 0.02" spacing (.5 mm); double the spacing because there are two tiers. The package connects these pads to the pin grid with 0.1" spacing (2.54 mm). Thus, the scale expands by about a factor of 2500 from the die's microscopic circuitry to the chip's pins. `
The ceramic package is manufactured through a complicated process.4 The process starts with flexible ceramic "green sheets", consisting of ceramic powder mixed with a binding agent. After holes for vias are created in the sheet, tungsten paste is silk-screened onto the sheet to form the wiring. The sheets are stacked, laminated under pressure, and then sintered at high temperature (1500ºC to 1600ºC) to create the rigid ceramic. The pins are brazed onto the bottom of the chip. Next, the pins and the inner contacts for the die are electroplated with gold.3 The die is mounted, gold bond wires are attached, and a metal cap is soldered over the die to encapsulate it. Finally, the packaged chip is tested, the package is labeled, and the chip is ready to be sold.
The diagram below shows a close-up of a signal layer inside the package. The pins are connected to the package's shelf pads through metal traces, spectacularly colored in the CT scan. (These traces are surprisingly wide and free-form; I expected narrower traces to reduce capacitance.) Bond wires connect the shelf pads to the bond pads on the silicon die. (The die image is added to the diagram; it is not part of the CT scan.) The large red circles are vias from the pins. Some vias connect to this signal layer, while other vias pass through to other layers. The smaller red circles are connections to a power layer; because the shelf pads are only on the two signal layers, the six power planes have connections to the signal layers for bonding. Since bond wires are only connected on the signal layers, the power layers need connections to pads on the signal layers.
The diagram below shows the corresponding portion of a power layer. A power layer looks completely different from a signal layer; it is a single conductive plane with holes. The grid of smaller holes allows the ceramic above and below this layer to bond, forming a solid piece of ceramic. The larger holes surround pin vias (red dots), allowing pin connections to pass through to a different layer. The red dots that contact the sheet are where power pins connect to this layer. Because the only connections to the die are from the signal layers, the power layers have connections to the signal layers; these are the smaller dots near the bond wires, either power vias passing through or vias connected to this layer.
With the JavaScript tool below, you can look at the package, layer by layer. Click on a radio button to select a layer. By observing the path of a pin through the layers, you can see where it ends up. For instance, the upper left pin passes through multiple layers until the upper signals layer connects it to the die. The pin to its right passes through all the layers until it reaches the logic Vcc plane on top. (Vcc is the 5-volt supply that powers the chip, called Vcc for historical reasons.)
If you select the logic Vcc plane above, you'll see a bright blotchy square in the center. This is not the die itself, I think, but the adhesive that attaches the die to the package, epoxy filled with silver to provide thermal and electrical conductivity. Since silver blocks X-rays, it is highly visible in the image.
What surprised me most about the scans was seeing wires that stick out to the sides of the package. These wires are used during manufacturing when the pins are electroplated with gold.5 In order to electroplate the pins, each pin must be connected to a negative voltage so it can function as a cathode. This is accomplished by giving each pin a separate wire that goes to the edge of the package.
This diagram below compares the CT scan (above) to a visual side view of the package (below). The wires are almost invisible, but can be seen as darker spots. The arrows show how three of these spots match with the CT scan; you can match up the other spots.6
According to the datasheet, the 386 has 20 pins connected to +5V power (Vcc) and 21 pins connected to ground (Vss). Studying the die, I noticed that the I/O circuitry in the 386 has separate power and ground connections from the logic circuitry. The motivation is that the output pins require high-current driver circuits. When a pin switches from 0 to 1 or vice versa, this can cause a spike on the power and ground wiring. If this spike is too large, it can interfere with the processor's logic, causing malfunctions. The solution is to use separate power wiring inside the chip for the I/O circuitry and for the logic circuitry, connected to separate pins. On the motherboard, these pins are all connected to the same power and ground, but decoupling capacitors absorb the I/O spikes before they can flow into the chip's logic.
The diagram below shows how the two power and ground networks look on the die, with separate pads and wiring. The square bond pads are at the top, with dark bond wires attached. The white lines are the two layers of metal wiring, and the darker regions are circuitry. Each I/O pin has a driver circuit below it, consisting of relatively large transistors to pull the pin high or low. This circuitry is powered by the horizontal lines for I/O Vcc (light red) and I/O ground (Vss, light blue). Underneath each I/O driver is a small logic circuit, powered by thinner Vcc (dark red) and Vss (dark blue). Thicker Vss and Vcc wiring goes to the logic in the rest of the chip. Thus, if the I/O circuitry causes power fluctuations, the logic circuit remains undisturbed, protected by its separate power wiring.
The datasheet doesn't mention the separate I/O and logic power networks, but by using the CT scans, I determined which pins power I/O, and which pins power logic. In the diagram below, the light red and blue pins are power and ground for I/O, while the dark red and blue pins are power and ground for logic. The pins are scattered across the package, allowing power to be supplied to all four sides of the die.
As the diagram above shows, the 386 has eight pins labeled "NC" (No Connect)—when the chip is installed in a computer, the motherboard must leave these pins unconnected. You might think that the 132-pin package simply has eight extra, unneeded pins, but it's more complicated than that. The photo below shows five bond pads at the bottom of the 386 die. Three of these pads have bond wires attached, but two have no bond wires: these correspond to No Connect pins. Note the black marks in the middle of the pads: the marks are from test probes that were applied to the die during testing.7 The No Connect pads presumably have a function during this testing process, providing access to an important internal signal.
Seven of the eight No Connect pads are almost connected: the package has a spot for a bond wire in the die cavity and the package has internal wiring to a No Connect pin. The only thing missing is the bond wire between the pad and the die cavity. Thus, by adding bond wires, Intel could easily create special chips with these pins connected, perhaps for debugging the test process itself.
The surprising thing is that one of the No Connect pads does have the bond wire in place, completing the connection to the external pin. (I marked this pin in green in the pinout diagram earlier.) From the circuitry on the die, this pin appears to be an output. If someone with a 386 chip hooks this pin to an oscilloscope, maybe they will see something interesting.
The earlier 8086 processor, for example, is packaged in a DIP (Dual-Inline Package) with two rows of pins. This makes it straightforward to figure out which pin (and thus which function) is connected to each pad on the die. However, since the 386 has a two-dimensional grid of pins, the mapping to the pads is unclear. You can guess that pins are connected to a nearby pad, but ambiguity remains. Without knowing the function of each pad, I have a harder time reverse-engineering the die.
In fact, my primary motivation for scanning the 386 package was to determine the pin-to-pad mapping and thus the function of each pad.8 Once I had the CT data, I was able to trace out each hidden connection between the pad and the external pin. The image below shows some of the labels; click here for the full, completely labeled image. As far as I know, this information hasn't been available outside Intel until now.
Intel's early processors were hampered by inferior packages, but by the time of the 386, Intel had realized the importance of packaging. In Intel's early days, management held the bizarre belief that chips should never have more than 16 pins, even though other companies used 40-pin packages. Thus, Intel's first microprocessor, the 4004 (1971), was crammed into a 16-pin package, limiting its performance. By 1972, larger memory chips forced Intel to move to 18-pin packages, extremely reluctantly.9 The eight-bit 8008 processor (1972) took advantage of this slightly larger package, but performance still suffered because signals were forced to share pins. Finally, Intel moved to the standard 40-pin package for the 8080 processor (1974), contributing to the chip's success. In the 1980s, pin-grid arrays became popular in the industry as chips required more and more pins. Intel used a ceramic pin grid array (PGA) with 68 pins for the 186 and 286 processors (1982), followed by the 132-pin package for the 386 (1985).
The main drawback of the ceramic package was its cost. According to the 386 oral history, the cost of the 386 die decreased over time to the point where the chip's package cost as much as the die. To counteract this, Intel introduced a low-cost plastic package for the 386 that cost just a dollar to manufacture, the Plastic Quad Flat Package (PQFP) (details).
In later Intel processors, the number of connections exponentially increased. A typical modern laptop processor uses a Ball Grid Array with 2049 solder balls; the chip is soldered directly onto the circuit board. Other Intel processors use a Land Grid Array (LGA): the chip has flat contacts called lands, while the socket has the pins. Some Xeon processors have 7529 contacts, a remarkable growth from the 16 pins of the Intel 4004.
From the outside, the 386's package looks like a plain chunk of ceramic. But the CT scan revealed surprising complexity inside, from numerous contacts for electroplating to six layers of wiring. Perhaps even more secrets lurk in the packages of modern processors.
Follow me on Bluesky (@righto.com), Mastodon (@kenshirriff@oldbytes.space), or RSS. (I've given up on Twitter.) Thanks to Jon Bruner and Lumafield for scanning the chip. Lumafield's interactive CT scan of the 386 package is available here if you to want to examine it yourself. Lumafield also scanned a 1960s cordwood flip-flop and the Soviet Globus spacecraft navigation instrument for us. Thanks to John McMaster for taking 2D X-rays.
I removed the metal lid with a chisel, as hot air failed to desolder the lid. A few pins were bent in the process, but I straightened them out, more or less. ↩
The 386 package is described in "High Performance Technology, Circuits and Packaging for the 80386", Proceedings, ICCD Conference, Oct. 1986. (Also see Design and Test of the 80386 by Pat Gelsinger, former Intel CEO.)
The paper gives the following requirements for the 386 package:
The first and second criteria motivated the selection of a 132-pin ceramic pin grid array (PGA). The custom six-layer package was designed to achieve the third objective. The power network is claimed to have an inductance of 4.5 nH per power pad on the device, compared to 12-14 nH for a standard package, about a factor of 3 better.
The paper states that logic Vcc, logic Vss, I/O Vcc, and I/O Vss each have 10 pins assigned. Curiously, the datasheet states that the 386 has 20 Vcc pins and 21 Vss pins, which doesn't add up. From my investigation, the "extra" pin is assigned to logic Vss, which has 11 pins. ↩
I estimate that the 386 package contains roughly 0.16 grams of gold, currently worth about $16. It's hard to find out how much gold is in a processor since online numbers are all over the place. Many people recover the gold from chips, but the amount of gold one can recover depends on the process used. Moreover, people tend to keep accurate numbers to themselves so they can profit. But I made some estimates after searching around a bit. One person reports 9.69g of gold per kilogram of chips, and other sources seem roughly consistent. A ceramic 386 reportedly weighs 16g. This works out to 160 mg of gold per 386. ↩
I don't have information on Intel's package manufacturing process specifically. This description is based on other descriptions of ceramic packages, so I don't guarantee that the details are correct for the 386. A Fujitsu patent, Package for enclosing semiconductor elements, describes in detail how ceramic packages for LSI chips are manufactured. IBM's process for ceramic multi-chip modules is described in Multi-Layer Ceramics Manufacturing, but it is probably less similar. ↩
An IBM patent, Method for shorting pin grid array pins for plating, describes the prior art of electroplating pins with nickel and/or gold. In particular, it describes using leads to connect all input/output pins to a common bus at the edge of the package, leaving the long leads in the structure. This is exactly what I see in the 386 chip. The patent mentions that a drawback of this approach is that the leads can act as antennas and produce signal cross-talk. Fujitsu patent Package for enclosing semiconductor elements also describes wires that are exposed at side surfaces. This patent covers methods to avoid static electricity damage through these wires. (Picking up a 386 by the sides seems safe, but I guess there is a risk of static damage.)
Note that each input/output pin requires a separate wire to the edge. However, the multiple pins for each power or ground plane are connected inside the package, so they do not require individual edge connections; one or two suffice. ↩
To verify that the wires from pins to the edges of the chip exist and are exposed, I used a multimeter and found connectivity between pins and tiny spots on the sides of the chip. ↩
To reduce costs, each die is tested while it is still part of the silicon wafer and each faulty die is marked with an ink spot. The wafer is "diced", cutting it apart into individual dies, and only the functional, unmarked dies are packaged, avoiding the cost of packaging a faulty die. Additional testing takes place after packaging, of course. ↩
I tried several approaches to determine the mapping between pads and pins before using the CT scan. I tried to beep out the connections between the pins and the pads with a multimeter, but because the pads are so tiny, the process was difficult, error-prone, and caused damage to the package.
I also looked at the pinout of the 386 in a plastic package (datasheet).
Since the plastic package has the pins in a single ring around the border, the mapping to the die is
straightforward.
Unfortunately, the 386 die was slightly redesigned at this time, so some pads were moved around and new pins
were added, such as FLT#.
It turns out that the pinout for the plastic chip almost matches the die I examined, but not quite. ↩
In his oral history, Federico Faggin, a designer of the 4004, 8008, and Z80 processors, describes Intel's fixation on 16-pin packages. When a memory chip required 18 pins instead of 16, it was "like the sky had dropped from heaven. I never seen so [many] long faces at Intel, over this issue, because it was a religion in Intel; everything had to be 16 pins, in those days. It was a completely silly requirements [sic] to have 16 pins." At the time, other manufacturers were using 40- and 48-pin packages, so there was no technical limitation, just a minor cost saving from the smaller package. ↩
Have you ever wanted to reverse engineer an analog chip from a die photo? Wanted to understand what's inside the "black box" of an integrated circuit? In this article, I explain my reverse engineering process, using the Philips TDA7000 FM radio receiver chip as an example. This chip was the first FM radio receiver on a chip.1 It was designed in 1977—an era of large transistors and a single layer of metal—so it is much easier to examine than modern chips. Nonetheless, the TDA7000 is a non-trivial chip with over 100 transistors. It includes common analog circuits such as differential amplifiers and current mirrors, along with more obscure circuits such as Gilbert cell mixers.
The die photo above shows the silicon die of the TDA7000; I've labeled the main functional blocks and some interesting components. Arranged around the border of the chip are 18 bond pads: the pads are connected by thin gold bond wires to the pins of the integrated circuit package. In this chip, the silicon appears greenish, with slightly different colors—gray, pink, and yellow-green—where the silicon has been "doped" with impurities to change its properties. Carefully examining the doping patterns will reveal the transistors, resistors, and other microscopic components that make up the chip.
The most visible part of the die is the metal wiring, the speckled white lines that connect the silicon structures. The metal layer is separated from the silicon underneath by an insulating oxide layer, allowing metal lines to pass over other circuitry without problem. Where a metal wire connects to the underlying silicon, a small white square is visible; this square is a hole in the oxide layer, allowing the metal to contact the silicon.
This chip has a single layer of metal, so it is much easier to examine than modern chips with a dozen or more layers of metal. However, the single layer of metal made it much more difficult for the designers to route the wiring while avoiding crossing wires. In the die photo above, you can see how the wiring meanders around the circuitry in the middle, going the long way since the direct route is blocked. Later, I'll discuss some of the tricks that the designers used to make the layout successful.
Transistors are the key components in a chip, acting as switches, amplifiers, and other active devices. While modern integrated circuits are fabricated from MOS transistors, earlier chips such as the TDA7000 were constructed from bipolar transistors: NPN and PNP transistors. The photo below shows an NPN transistor in the TDA7000 as it appears on the chip. The different shades are regions of silicon that have been doped with various impurities, forming N and P regions with different electrical properties. The white lines are the metal wiring connected to the transistor's collector (C), emitter (E), and base (B). Below the die photo, the cross-section diagram shows how the transistor is constructed. The region underneath the emitter forms the N-P-N sandwich that defines the NPN transistor.
The parts of an NPN transistor can be identified by their appearance. The emitter is a compact spot, surrounded by the gray silicon of the base region. The collector is larger and separated from the emitter and base, sometimes separated by a significant distance. The colors may appear different in other chips, but the physical structures are similar. Note that although the base is in the middle conceptually, it is often not in the middle of the physical layout.
The transistor is surrounded by a yellowish-green border of P+ silicon; this border is an important part of the structure because it isolates the transistor from neighboring transistors.2 The isolation border is helpful for reverse-engineering because it indicates the boundaries between transistors.
You might expect PNP transistors to be similar to NPN transistors, just swapping the roles of N and P silicon. But for a variety of reasons, PNP transistors have an entirely different construction. They consist of a circular emitter (P), surrounded by a ring-shaped base (N), which is surrounded by the collector (P). This forms a P-N-P sandwich horizontally (laterally), unlike the vertical structure of an NPN transistor. In most chips, distinguishing NPN and PNP transistors is straightforward because NPN transistors are rectangular while PNP transistors are circular.
The diagram above shows one of the PNP transistors in the TDA7000. As with the NPN transistor, the emitter is a compact spot. The collector consists of gray P-type silicon; in contrast, the base of an NPN transistor consists of gray P-type silicon. Moreover, unlike the NPN transistor, the base contact of the PNP transistor is at a distance, while the collector contact is closer. (This is because most of the silicon inside the isolation boundary is N-type silicon. In a PNP transistor, this region is connected to the base, while in an NPN transistor, this region is connected to the collector.)
It turns out that PNP transistors have poorer performance than NPN transistors for semiconductor reasons3, so most analog circuits use NPN transistors except when PNP transistors are necessary. For instance, the TDA7000 has over 100 NPN transistors but just nine PNP transistors. Accordingly, I'll focus my discussion on NPN transistors.
Resistors are a key component of analog chips. The photo below shows a zig-zagging resistor in the TDA7000, formed from gray P-type silicon. The resistance is proportional to the length,4 so large-valued resistors snake back and forth to fit into the available space. The two red arrows indicate the contacts between the ends of the resistor and the metal wiring. Note the isolation region around the resistor, the yellowish border. Without this isolation, two resistors (formed of P-silicon) embedded in N-silicon could form an unintentional PNP transistor.
Unfortunately, resistors in ICs are very inaccurate; the resistances can vary by 50% from chip to chip. As a result, analog circuits are typically designed to depend on the ratio of resistor values, which is fairly constant within a chip. Moreover, high-value resistors are inconveniently large. We'll see below some techniques to reduce the need for large resistances.
Capacitors are another important component in analog circuits. The capacitor below is a "junction capacitor", which uses a very large reverse-biased diode as a capacitor. The pink "fingers" are N-doped regions, embedded in the gray P-doped silicon. The fingers form a "comb capacitor"; this layout maximizes the perimeter area and thus increases the capacitance. To produce the reverse bias, the N-silicon fingers are connected to the positive voltage supply through the upper metal strip. The P silicon is connected to the circuit through the lower metal strip.
How does a diode junction form a capacitor? When a diode is reverse-biased, the contact region between N and P silicon becomes "depleted", forming a thin insulating region between the two conductive silicon regions. Since an insulator between two conducting surfaces forms a capacitor, the diode acts as a capacitor. One problem with a diode capacitor is that the capacitance varies with the voltage because the thickness of the depletion region changes with voltage. But as we'll see later, the TDA7000's tuning circuit turns this disadvantage into a feature.
Other chips often create a capacitor with a plate of metal over silicon, separated by a thin layer of oxide or other dielectric. However, the manufacturing process for bipolar chips generally doesn't provide thin oxide, so junction capacitors are a common alternative.5 On-chip capacitors take up a lot of space and have relatively small capacitance, so IC designers try to avoid capacitors. The TDA7000 has seven on-chip capacitors but most of the capacitors in this design are larger, external capacitors: the chip uses 12 of its 18 pins just to connect external capacitors to the necessary points in the internal circuitry.
A few circuits are very common in analog chips. In this section, I'll explain some of these circuits, but first, I'll give a highly simplified explanation of an NPN transistor, the minimum you should know for reverse engineering. (PNP transistors are similar, except the polarities of the voltages and currents are reversed. Since PNP transistors are rare in the TDA7000, I won't go into details.)
In a transistor, the base controls the current between the collector and the emitter, allowing the transistor to operate as a switch or an amplifier. Specifically, if a small current flows from the base of an NPN transistor to the emitter, a much larger current can flow from the collector to the emitter, larger, perhaps, by a factor of 100.6 To get a current to flow, the base must be about 0.6 volts higher than the emitter. As the base voltage continues to increase, the base-emitter current increases exponentially, causing the collector-emitter current to increase. (Normally, a resistor will ensure that the base doesn't get much more than 0.6V above the emitter, so the currents stay reasonable.)
NPN transistor circuits have some general characteristics. When there is no base current, the transistor is off: the collector is high and the emitter is low. When the transistor turns on, the current through the transistor pulls the collector voltage lower and the emitter voltage higher. Thus, in a rough sense, the emitter is the non-inverting output and the collector is the inverting output.
The complete behavior of transistors is much more complicated. The nice thing about reverse engineering is that I can assume that the circuit works: the designers needed to consider factors such as the Early effect, capacitance, and beta, but I can ignore them.
One of the simplest transistor circuits is the emitter follower. In this circuit, the emitter voltage follows the base voltage, staying about 0.6 volts below the base. (The 0.6 volt drop is also called a "diode drop" because the base-emitter junction acts like a diode.)
This behavior can be explained by a feedback loop. If the emitter voltage is too high, the current from the base to the emitter drops, so the current through the collector drops due to the transistor's amplification. Less current through the resistor reduces the voltage across the resistor (from Ohm's Law), so the emitter voltage goes down. Conversely, if the emitter voltage is too low, the base-emitter current increases, increasing the collector current. This increases the voltage across the resistor, and the emitter voltage goes up. Thus, the emitter voltage adjusts until the circuit is stable; at this point, the emitter is 0.6 volts below the base.
You might wonder why an emitter follower is useful. Although the output voltage is lower, the transistor can supply a much higher current. That is, the emitter follower amplifies a weak input current into a stronger output current. Moreover, the circuitry on the input side is isolated from the circuitry on the output side, preventing distortion or feedback.
Most analog chips make extensive use of a circuit called a current mirror. The idea is you start with one known current, and then you can "clone" multiple copies of the current with a simple transistor circuit, the current mirror.
In the following circuit, a current mirror is implemented with two identical PNP transistors. A reference current passes through the transistor on the right. (In this case, the current is set by the resistor.) Since both transistors have the same emitter voltage and base voltage, they source the same current, so the current on the left matches the reference current (more or less).7
A common use of a current mirror is to replace resistors. As mentioned earlier, resistors inside ICs are inconveniently large. It saves space to use a current mirror instead of multiple resistors whenever possible. Moreover, the current mirror is relatively insensitive to the voltages on the different branches, unlike resistors. Finally, by changing the size of the transistors (or using multiple collectors of different sizes), a current mirror can provide different currents.
The TDA7000 doesn't use current mirrors as much as I'd expect, but it has a few. The die photo above shows one of its current mirrors, constructed from PNP transistors with their distinctive round appearance. Two important features will help you recognize a current mirror. First, one transistor has its base and collector connected; this is the transistor that controls the current. In the photo, the transistor on the right has this connection. Second, the bases of the two transistors are connected. This isn't obvious above because the connection is through the silicon, rather than in the metal. The trick is that these PNP transistors are inside the same isolation region. If you look at the earlier cross-section of a PNP transistor, the whole N-silicon region is connected to the base. Thus, two PNP transistors in the same isolation region have their bases invisibly linked, even though there is just one base contact from the metal layer.
Analog circuits frequently need a constant current. A straightforward approach is to use a resistor; if a constant voltage is applied, the resistor will produce a constant current. One disadvantage is that circuits can cause the voltage to vary, generating unwanted current fluctuations. Moreover, to produce a small current (and minimize power consumption), the resistor may need to be inconveniently large. Instead, chips often use a simple circuit to control the current: this circuit is called a "current sink" if the current flows into it and a "current source" if the current flows out of it.
Many chips use a current mirror as a current source or sink instead. However, the TDA7000 uses a different approach: a transistor, a resistor, and a reference voltage.8 The transistor acts like an emitter follower, causing a fixed voltage across the resistor. By Ohm's Law, this yields a fixed current. Thus, the circuit sinks a fixed current, controlled by the reference voltage and the size of the resistor. By using a low reference voltage, the resistor can be kept small.
If you see two transistors with the emitters connected, chances are that it is a differential amplifier: the most common two-transistor subcircuit used in analog ICs.9 The idea of a differential amplifier is that it takes the difference of two inputs and amplifies the result. The differential amplifier is the basis of the operational amplifier (op amp), the comparator, and other circuits. The TDA7000 uses multiple differential pairs for amplification. For filtering, the TDA7000 uses op-amps, formed from differential amplifiers.10
The schematic below shows a simple differential pair. The current sink at the bottom provides a fixed current I, which is split between the two input transistors. If the input voltages are equal, the current will be split equally into the two branches (I1 and I2). But if one of the input voltages is a bit higher than the other, the corresponding transistor will conduct more current, so that branch gets more current and the other branch gets less. The resistors in each branch convert the current to a voltage; either side can provide the output. A small difference in the input voltages results in a large output voltage, providing the amplification. (Alternatively, both sides can be used as a differential output, which can be fed into a second differential amplifier stage to provide more amplification. Note that the two branches have opposite polarity: when one goes up, the other goes down.)
The diagram below shows the locations of differential amps, voltage references, mixers, and current mirrors. As you can see, these circuits are extensively used in the TDA7000.
Over the years, I've found various techniques helpful for tracing out the circuitry in an IC. In this section, I'll describe some of those techniques.
First, take a look at the datasheet if available. In the case of the TDA7000, the datasheet and application note provide a detailed block diagram and a description of the functionality.21 Sometimes datasheets include a schematic of the chip, but don't be too trusting: datasheet schematics are often simplified. Moreover, different manufacturers may use wildly different implementations for the same part number. Patents can also be helpful, but they may be significantly different from the product.
Mapping the pinout in the datasheet to the pads on the die will make reverse engineering much easier. The power and ground pads are usually distinctive, with thick traces that go to all parts of the chip, as shown in the photo below. Once you have identified the power and ground pads, you can assign the other pads in sequence from the datasheet. Make sure that these pad assignments make sense. For instance, the TDA7000 datasheet shows special circuitry between pads 5 and 6 and between pads 13 and 14; the corresponding tuning diodes and RF transistors are visible on the die. In most chips, you can distinguish output pins by the large driver transistors next to the pad, but this turns out not to help with the TDA7000. Finally, note that chips sometimes have test pads that don't show up in the datasheet. For instance, the TDA7000 has a test pad, shown below; you can tell that it is a test pad because it doesn't have a bond wire.
Once I've determined the power and ground pads, I trace out all the power and ground connections on the die. This makes it much easier to understand the circuits and also avoids the annoyance of following a highly-used signal around the chip only to discover that it is simply ground. Note that NPN transistors will have many collectors connected to power and emitters connected to ground, perhaps through resistors. If you find the opposite situation, you probably have power and ground reversed.
For a small chip, a sheet of paper works fine for sketching out the transistors and their connections. But with a larger chip, I find that more structure is necessary to avoid getting mixed up in a maze of twisty little wires, all alike. My solution is to number each component and color each wire as I trace it out, as shown below. I use the program KiCad to draw the schematic, using the same transistor numbering. (The big advantage of KiCad over paper is that I can move circuits around to get a nicer layout.)
It works better to trace out the circuitry one area at a time, rather than chasing signals all over the chip. Chips are usually designed with locality, so try to avoid following signals for long distances until you've finished up one block. A transistor circuit normally needs to be connected to power (if you follow the collectors) and ground (if you follow the emitters).11 Completing the circuit between power and ground is more likely to give you a useful functional block than randomly tracing out a chain of transistors. (In other words, follow the bases last.)
Finally, I find that a circuit simulator such as LTspice is handy when trying to understand the behavior of mysterious transistor circuits. I'll often whip up a simulation of a small sub-circuit if its behavior is unclear.
Before I explain how the TDA7000 chip works, I'll give some background on FM (Frequency Modulation). Suppose you're listening to a rock song on 97.3 FM. The number means that the radio station is transmitting at a carrier frequency of 97.3 megahertz. The signal, perhaps a Beyoncé song, is encoded by slightly varying the frequency, increasing the frequency when the signal is positive and decreasing the frequency when the signal is negative. The diagram below illustrates frequency modulation; the input signal (red) modulates the output. Keep in mind that the modulation is highly exaggerated in the diagram; the modulation would be invisible in an accurate diagram since a radio broadcast changes the frequency by at most ±75 kHz, less than 0.1% of the carrier frequency.
FM radio's historical competitor is AM (Amplitude Modulation), which varies the height of the signal (the amplitude) rather than the frequency.12 One advantage of FM is that it is more resistant to noise than AM; an event such as lightning will interfere with the signal amplitude but will not change the frequency. Moreover, FM radio provides stereo, while AM radio is mono, but this is due to the implementation of radio stations, not a fundamental characteristic of FM versus AM. (The TDA7000 chip doesn't implement stereo.13) Due to various factors, FM stations require more bandwidth than AM, so FM stations are spaced 200 kHz apart while AM stations are just 10 kHz apart.
An FM receiver such as the TDA7000 must demodulate the radio signal to recover the transmitted audio, converting the changing frequency into a changing signal level. FM is more difficult to demodulate than AM, which can literally be done with a piece of rock: lead sulfide in a crystal detector. There are several ways to implement an FM demodulator; this chip uses a technique called a quadrature detector. The key to a quadrature detector is a circuit that shifts the phase, with the amount of phase shift depending on the frequency. The detector shifts the signal by approximately 90º, multiplies it by the original signal, and then smooths it out with a low-pass filter. If you do this with a sine wave and a 90º phase shift, the result turns out to be 0. But since the phase shift depends on the frequency, a higher frequency gets shifted by more than 90º while a lower frequency gets shifted by less than 90º. The final result turns out to be approximately linear with the frequency, positive for higher frequencies and negative for lower frequencies. Thus, the FM signal is converted into the desired audio signal.
Like most radios, the TDA7000 uses a technique called superheterodyning that was invented around 1917. The problem is that FM radio stations use frequencies from 88.0 MHz to 108.0 MHz. These frequencies are too high to conveniently handle on a chip. Moreover, it is difficult to design a system that can process a wide range of frequencies. The solution is to shift the desired radio station's signal to a frequency that is fixed and much lower. This frequency is called the intermediate frequency. Although FM radios commonly use an intermediate frequency of 10.7 MHz, this was still too high for the TDA7000, so the designers used an intermediate frequency of just 70 kilohertz. This frequency shift is accomplished through superheterodyning.
For example, suppose you want to listen to the radio station at 97.3 MHz. When you tune to this station, you are actually tuning the local oscillator to a frequency that is 70 kHz lower, 97.23 MHz in this case. The local oscillator signal and the radio signal are mixed by multiplying them. If you multiply two sine waves, you get one sine wave at the difference of the frequencies and another sine wave at the sum of the frequencies. In this case, the two signals are at 70 kHz and 194.53 MHz. A low-pass filter (the IF filter) discards everything above 70 kHz, leaving just the desired radio station, now at a fixed and conveniently low frequency. The rest of the radio can then be optimized to work at 70 kHz.
But how do you multiply two signals? This is accomplished with a circuit called a Gilbert cell.14 This circuit takes two differential inputs, multiplies them, and produces a differential output. The Gilbert cell is a bit tricky to understand,15 but you can think of it as a stack of differential amplifiers, with the current directed along one of four paths, depending on which transistors turn on. For instance, if the A and B inputs are both positive, current will flow through the leftmost transistor, labeled "pos×pos". Likewise, if the A and B inputs are both negative, current flows through the rightmost transistor, labeled "neg×neg". The outputs from both transistors are connected, so both cases produce a positive output. Conversely, if one input is positive and the other is negative, current flows through one of the middle transistors, producing a negative output. Since the multiplier handles all four cases of positive and negative inputs, it is called a "four-quadrant" multiplier.
Although the Gilbert cell is an uncommon circuit in general, the TDA7000 uses it in multiple places. The first mixer implements the superheterodyning. A second mixer provides the FM demodulation, multiplying signals in the quadrature detector described earlier. The TDA7000 also uses a mixer for its correlator, which determines if the chip is tuned to a station or not.16 Finally, a Gilbert cell switches the audio off when the radio is not properly tuned. On the die, the Gilbert cell has a nice symmetry that reflects the schematic.
One of the trickiest parts of the TDA7000 design is how it manages to use an intermediate frequency of just 70 kilohertz. The problem is that broadcast FM has a "modulation frequency deviation" of 75 kHz, which means that the broadcast frequency varies by up to ±75 kHz. The mixer shifts the broadcast frequency down to 70 kHz, but the shifted frequency will vary by the same amount as the received signal. How can you have a 70 kilohertz signal that varies by 75 kilohertz? What happens when the frequency goes negative?
The solution is that the local oscillator frequency (i.e., the frequency that the radio is tuned to) is continuously modified to track the variation in the broadcast frequency. Specifically, a change in the received frequency causes the local oscillator frequency to change, but only by 80% as much. For instance, if the received frequency decreases by 5 hertz, the local oscillator frequency is decreased by 4 hertz. Recall that the intermediate frequency is the difference between the two frequencies, generated by the mixer, so the intermediate frequency will decrease by just 1 hertz, not 5 hertz. The result is that as the broadcast frequency changes by ±75 kHz, the local oscillator frequency changes by just ±15 kHz, so it never goes negative.
How does the radio constantly adjust the frequency? The fundamental idea of FM is that the frequency shift corresponds to the output audio signal. Since the output signal tracks the frequency change, the output signal can be used to modify the local oscillator's frequency, using a voltage-controlled oscillator.17 Specifically, the circuit uses special "varicap" diodes that vary their capacitance based on the voltage that is applied. As described earlier, the thickness of a diode's "depletion region" depends on the voltage applied, so the diode's capacitance will vary with voltage. It's not a great capacitor, but it is good enough to adjust the frequency.
The image above shows how these diodes appear on the die. The diodes are relatively large and located between two bond pads. The two diodes have interdigitated "fingers"; this increases the capacitance as described earlier with the "comb capacitor". The slightly grayish "background" region is the P-type silicon, with a silicon control line extending to the right. (Changing the voltage on this line changes the capacitance.) Regions of N-type silicon are underneath the metal fingers, forming the PN junctions of the diodes.
Keep in mind that most of the radio tuning is performed with a variable capacitor that is external to the chip and adjusts the frequency from 88 MHz to 108 MHz. The capacitance of the diodes provides the much smaller adjustment of ±60 kHz. Thus, the diodes only need to provide a small capacitance shift.
The VCO and diodes will also adjust the frequency to lock onto the station if the tuning is off by a moderate amount, say, 100 kHz. However, if the tuning is off by a large amount, say, 200 kHz, the FM detector has a "sideband" and the VCO can erroneously lock onto this sideband. This is a problem because the sideband is weak and nonlinear so reception will be bad and will have harmonic distortion. To avoid this problem, the correlator will detect that the tuning is too far off (i.e. the local oscillator is way off from 70 kHz) and will replace the audio with white noise. Thus, the user will realize that they aren't on the station and adjust the tuning, rather than listening to distorted audio and blaming the radio.
Where does the radio get the noise signal to replace distorted audio? The noise is generated from the circuit below, which uses the thermal noise from diodes, amplified by a differential amplifier. Specifically, each side of the differential amplifier is connected to two transistors that are wired as diodes (using the base-emitter junction). Random thermal fluctuations in the transistors will produce small voltage changes on either side of the amplifier. The amplifier boosts these fluctuations, creating the white noise output.
Because this chip has just one layer of metal, the designers had to go to considerable effort to connect all the components without wires crossing. One common technique to make routing easier is to separate a transistor's emitter, collector, and base, allowing wires to pass over the transistor. The transistor below is an example. Note that the collector, base, and emitter have been stretched apart, allowing one wire to pass between the collector and the base, while two more pass between the base and the emitter. Moreover, the transistor layout is flexible: this one has the base in the middle, while many others have the emitter in the middle. (Putting the collector in the middle won't work since the base needs to be next to the emitter.)
The die photo below illustrates a few more routing tricks. This photo shows one collector, three emitters, and four bases, but there are three transistors. How does that work? First, these three transistors are in the same isolation region, so they share the same "tub" of N-silicon. If you look back at the cross-section of an NPN transistor, you'll see that this tub is connected to the collector contact. Thus, all three transistors share the same collector.18 Next, the two bases on the left are connected to the same gray P-silicon. Thus, the two base contacts are connected and function as a single base. In other words, this is a trick to connect the two base wires together through the silicon, passing under the four other metal wires in the way. Finally, the two transistors on the right have the emitter and base slightly separated so a wire can pass between them. When reverse-engineering a chip, be on the lookout for unusual transistor layouts such as these.
When all else failed, the designers could use a "cross-under" to let a wire pass under other wires. The cross-under is essentially a resistor with a relatively low resistance, formed from N-type silicon (pink in the die photo below). Because silicon has much higher resistance than metal, cross-unders are avoided unless necessary. I see just two cross-unders in the TDA7000.
The circuit that caused me the most difficulty is the noise generator below. The transistor highlighted in red below looks straightforward: a resistor is connected to the collector, which is connected to the base. However, the transistor turned out to be completely different: the collector (red arrow) is on the other side of the circuit and this collector is shared with five other transistors. The structure that I thought was the collector is simply the contact at the end of the resistor, connected to the base.
The TDA7000 almost didn't become a product. It was invented in 1977 by two engineers at the Philips research labs in the Netherlands. Although Philips was an innovative consumer electronics company in the 1970s, the Philips radio group wasn't interested in an FM radio chip. However, a rogue factory manager built a few radios with the chips and sent them to Japanese companies. The Japanese companies loved the chip and ordered a million of them, convincing Philips to sell the chips.
The TDA7000 became a product in 1983—six years after its creation—and reportedly more than 5 billion have now been sold.19 Among other things, the chip allowed an FM radio to be built into a wristwatch, with the headphone serving as an antenna. Since the TDA7000 vastly simplified the construction of a radio, the chip was also popular with electronics hobbyists. Hobbyist magazines provided plans and the chip could be obtained from Radio Shack.20
Why reverse engineer a chip such as the TDA7000? In this case, I was answering some questions for the IEEE microchips exhibit, but even when reverse engineering isn't particularly useful, I enjoy discovering the logic behind the mysterious patterns on the die. Moreover, the TDA7000 is a nice chip for reverse engineering because it has large features that are easy to follow, but it also has many different circuits. Since the chip has over 100 transistors, you might want to start with a simpler chip, but the TDA7000 is a good exercise if you want to increase your reverse-engineering skills. If you want to check your results, my schematic of the TDA7000 is here; I don't guarantee 100% accuracy :-) In any case, I hope you have enjoyed this look at reverse engineering.
Follow me on Bluesky (@righto.com), Mastodon (@kenshirriff@oldbytes.space), or RSS. (I've given up on Twitter.) Thanks to Daniel Mitchell for asking me about the TDA7000 and providing the die photo; be sure to check out the IEEE Chip Hall of Fame's TDA7000 article.
The first "radio-on-a-chip" was probably the Ferranti ZN414 from 1973, which implemented an AM radio. An AM radio receiver is much simpler than an FM receiver (you really just need a diode), explaining why the AM radio ZN414 was a decade earlier than the FM radio TDA7000. As a 1973 article stated, "There are so few transistors in most AM radios that set manufacturers see little profit in developing new designs around integrated circuits merely to shave already low semiconductor costs." The ZN414 has just three pins and comes in a plastic package resembling a transistor. The ZN414 contains only 10 transistors, compared to about 132 in the TDA7000. ↩
The transistors are isolated by the P+ band that surrounds them. Because this band is tied to ground, it is at a lower voltage than the neighboring N regions. As a result, the PN border between transistor regions acts as a reverse-biased diode PN junction and current can't flow. (For current to flow, the P region must be positive and the N region must be negative.)
The invention of this isolation technique was a key step in making integrated circuits practical. In earlier integrated circuits, the regions were physically separated and the gaps were filled with non-conductive epoxy. This manufacturing process was both difficult and unreliable. ↩
NPN transistors perform better than PNP transistors due to semiconductor physics. Specifically, current in NPN transistors is primarily carried by electrons, while current in PNP transistors is primarily carried by "holes", the positively-charged absence of an electron. It turns out that electrons travel better in silicon than holes—their "mobility" is higher.
Moreover, the lateral construction of a PNP transistor results in a worse transistor than the vertical construction of an NPN transistor. Why can't you just swap the P and N domains to make a vertical PNP transistor? The problem is that the doping elements aren't interchangeable: boron is used to create P-type silicon, but it diffuses too rapidly and isn't soluble enough in silicon to make a good vertical PNP transistor. (See page 280 of The Art of Analog Layout for details). Thus, ICs are designed to use NPN transistors instead of PNP transistors as much as possible. ↩
The resistance of a silicon resistor is proportional to its length divided by its width. (This makes sense since increasing the length is like putting resistors in series, while increasing the width is like putting resistors in parallel.) When you divide length by width, the units cancel out, so the resistance of silicon is described with the curious unit ohms per square (Ω/□). (If a resistor is 5 mm long and 1 mm wide, you can think of it as five squares in a chain; the same if it is 5 µm by 1 µm. It has the same resistance in both cases.)
A few resistances are mentioned on the TDA7000 schematic in the datasheet. By measuring the corresponding resistors on the die, I calculate that the resistance on the die is about 200 ohms per square (Ω/□). ↩
See The Art of Analog Layout page 197 for more information on junction capacitors. ↩
You might wonder about the names "emitter" and "collector"; it seems backward that current flows from the collector to the emitter. The reason is that in an NPN transistor, the emitter emits electrons, they flow to the collector, and the collector collects them. The confusion arises because Benjamin Franklin arbitrarily stated that current flows from positive to negative. Unfortunately this "conventional current" flows in the opposite direction from the actual electrons. On the other hand, a PNP transistor uses holes—the absence of electrons—to transmit current. Positively-charged holes flow from the PNP transistor's emitter to the collector, so the flow of charge carriers matches the "conventional current" and the names "emitter" and "collector" make more sense. ↩
The basic current mirror circuit isn't always accurate enough. The TDA7000's current mirrors improve the accuracy by adding emitter degeneration resistors. Other chips use additional transistors for accuracy; some circuits are here. ↩
The reference voltages are produced with versions of the circuit below, with the output voltage controlled by the resistor values. In more detail, the bottom transistor is wired as a diode, providing a voltage drop of 0.6V. Since the upper transistor acts as an emitter follower, its base "should" be at 1.2V. The resistors form a feedback loop with the base: the current (I) will adjust until the voltage drop across R1 yields a base voltage of 1.2V. The fixed current (I) through the circuit produces a voltage drop across R1 and R2, determining the output voltage. (This circuit isn't a voltage regulator; it assumes that the supply voltage is stable.)
Note that this circuit will produce a reference voltage between 0.6V and 1.2V. Without the lower transistor, the voltage would be below 0.6V, which is too low for the current sink circuit. A closer examination of the circuit shows that the output voltage depends on the ratio between the resistances, not the absolute resistances. This is beneficial since, as explained earlier, resistors on integrated circuits have inaccurate absolute resistances, but the ratios are much more constant. ↩
Differential pairs are also called long-tailed pairs. According to Analysis and Design of Analog Integrated Circuits, differential pairs are "perhaps the most widely used two-transistor subcircuits in monolithic analog circuits." (p214)
Note that the transistors in the differential pair act like an emitter follower controlled by the higher input. That is, the emitters will be 0.6 volts below the higher base voltage. This is important since it shuts off the transistor with the lower base. (For example, if you put 2.1 volts in one base and 2.0 volts in the other base, you might expect that the base voltages would turn both transistors on. But the emitters are forced to 1.5 volts (2.1 - 0.6). The base-emitter voltage of the second transistor is now 0.5 volts (2.0 - 1.5), which is not enough to turn the transistor on.) ↩
Filters are very important to the TDA7000 and these filters are implemented by op-amps. If you want details, take a look at the application note, which describes the "second-order low-pass Sallen-Key" filter, first-order high-pass filter, active all-pass filter, and other filters. ↩
Most transistor circuits connect (eventually) to power and ground. One exception is open-collector outputs or other circuits with a pull-up resistor outside the chip. ↩
Nowadays, satellite radio such as SiriusXM provides another competitor to FM radio. SiriusXM uses QPSK (Quadrature Phase-Shift Keying), which encodes a digital signal by encoding pairs of bits using one of four different phase shifts. ↩
FM stereo is broadcast in a clever way that allows it to be backward-compatible with mono FM receivers. Specifically, the mono signal consists of the sum of the left and right channels, so you hear both channels combined. For stereo, the difference between the channels is also transmitted: the left channel minus the right channel. Adding this to the mono signal gives you the desired left channel, while subtracting this from the mono signal gives you the desired right channel. This stereo signal is shifted up in frequency using a somewhat tricky modulation scheme, occupying the audio frequency range from 23 kHz to 53 kHz, while the mono signal occupies the range 0 kHz to 15 kHz. (Note: these channels are combined to make an audio-frequency signal before the frequency modulation.) A mono FM receiver uses a low-pass filter to strip out the stereo signal so you hear the mono channel, while a stereo FM receiver has the circuitry to shift the stereo signal down and then add or subtract it. A later chip, the TDA7021T, supported a stereo signal, although it required a separate stereo decoder chip (TDA7040T)to generate the left and right channels. ↩
A while ago, I wrote about the Rockwell RC4200 analog multiplier chip. It uses a completely different technique from the Gilbert cell, essentially adding logarithms to perform multiplication. ↩
For a detailed explanation of the Gilbert cell, see Gilbert cell mixers. ↩
The TDA7000's correlator determines if the radio is correctly tuned or not. The idea is to multiply the signal by the signal delayed by half a cycle (180º) and inverted. If the signal is valid, the two signals match, giving a uniformly positive product. But if the frequency is off, the delay will be off, the signals won't match, and the product will be lower. Likewise, if the signal is full of noise, the signals won't match.
If the radio is mistuned, the audio is muted: the correlator provides the mute control signal. Specifically, when tuned properly, you hear the audio output, but when not tuned, the audio is replaced with a white noise signal, providing an indication that the tuning is wrong. The muting is accomplished with a Gilbert cell, but in a slightly unusual way. Instead of using differential inputs, the output audio is fed into one input branch and a white noise signal is fed into the other input branch. The mute control signal is fed into the upper transistors, selecting either the audio or the white noise. You can think of it as multiplying by +1 to get the audio and multiplying by -1 to get the noise. ↩
The circuit to track the frequency is called a Frequency-Locked Loop; it is analogous to a Phase-Locked Loop, except that the phase is not tracked. ↩
Some chips genuinely have transistors with multiple collectors, typically PNP transistors in current mirrors to produce multiple currents. Often these collectors have different sizes to generate different currents. NPN transistors with multiple emitters are used in TTL logic gates, while NPN transistors with multiple collectors are used in Integrated Injection Logic, a short-lived logic family from the 1970s. ↩
The history of the TDA7000 is based on the IEEE Spectrum article Chip Hall of Fame: Philips TDA7000 FM Receiver. Although the article claims that "more than 5 billion TDA7000s and variants have been sold", I'm a bit skeptical since that is more than the world's population at the time. Moreover, this detailed page on the TDA7000 states that the TDA7000 "found its way into a very few commercially made products". ↩
The TDA7000 was sold at stores such as Radio Shack; the listing below is from the 1988 catalog.
The TDA7000 is well documented, including the datasheet, application note, a technical review, an article, and Netherlands and US patents.
The die photo is from IEEE Microchips that Shook the World and the history is from IEEE Chip Hall of Fame: Philips TDA7000 FM Receiver. The Cool386 page on the TDA7000 has collected a large amount of information and is a useful resource.
The application note has a detailed block diagram, which makes reverse engineering easier:
If you're interested in analog chips, I highly recommend the book Designing Analog Chips, written by Hans Camenzind, the inventor of the famous 555 timer. The free PDF is here or get the book.
Back in 2021, a collector friend of ours was visiting a dusty warehouse in search of Apollo-era communications equipment. A box with NASA-style lights caught his eye—the "AGC Confirm" light suggested a connection with the Apollo Guidance Computer. Disappointingly, the box was just an empty chassis and the circuit boards were all missing. He continued to poke around the warehouse when, to his surprise, he found a bag on the other side of the warehouse that contained the missing boards! After reuniting the box with its wayward circuit cards, he brought it to us: could we make this undocumented unit work?
A label on the back indicated that it is an "Up-Data Link Confidence Test Set", built by Motorola. As the name suggests, the box was designed to test Apollo's Up-Data Link (UDL), a system that allowed digital commands to be sent up to the spacecraft. As I'll explain in detail below, these commands allowed ground stations to switch spacecraft circuits on or off, interact with the Apollo Guidance Computer, or set the spacecraft's clock. The Up-Data Link needed to be tested on the ground to ensure that its functions operated correctly. Generating the test signals for the Up-Data Link and verifying its outputs was the responsibility of the Up-Data Link Confidence Test Set (which I'll call the Test Set for short)
The Test Set illustrates how, before integrated circuits, complicated devices could be constructed from thumb-sized encapsulated modules. Since I couldn't uncover any documentation on these modules, I had to reverse-engineer them, discovering that different modules implemented everything from flip-flops and logic gates to opto-isolators and analog circuits. With the help of a Lumafield 3-dimensional X-ray scanner, we looked inside the modules and examined the discrete transistors, resistors, diodes, and other components mounted inside.
Reverse-engineering this system—from the undocumented modules to the mess of wiring—was a challenge. Mike found one NASA document that mentioned the Test Set, but the document was remarkably uninformative.1 Moreover, key components of the box were missing, probably removed for salvage years ago. In this article, I'll describe how we learned the system's functionality, uncovered the secrets of the encapsulated modules, built a system to automatically trace the wiring, and used the UDL Test Set in a large-scale re-creation of the Apollo communications system.
Before describing the Up-Data Link Test Set, I'll explain the Up-Data Link (UDL) itself. The Up-Data Link provided a mechanism for the Apollo spacecraft to receive digital commands from ground stations. These commands allowed ground stations to control the Apollo Guidance Computer, turn equipment on or off, or update the spacecraft's clock. Physically, the Up-Data Link is a light blue metal box with an irregular L shape, weighing almost 20 pounds.
The Apollo Command Module was crammed with boxes of electronics, from communication and navigation to power and sequencing. The Up-Data Link was mounted above the AC power inverters, below the Apollo Guidance Computer, and to the left of the waste management system and urine bags.
The Up-Data Link supported four types of messages:
Mission Control had direct access to the Apollo Guidance Computer (AGC) through the UDL, controlling the computer, keypress by keypress. That is, each message caused the UDL to simulate a keypress on the Display/Keyboard (DSKY), the astronaut's interface to the computer.
The spacecraft had a clock, called the Central Timing Equipment or CTE, that tracked the elapsed time of the mission, from days to seconds. A CTE message could set the clock to a specified time.
A system called Real Time Control (RTC) allowed the UDL to turn relays on or off, so some spacecraft systems to be controlled from the ground.2 These 32 relays, mounted inside the Up-Data Link box, could do everything from illuminating an Abort light—indicating that Mission Control says to abort—to controlling the data tape recorder or the S-band radio.
Finally, the UDL supported two test messages to "exercise all process, transfer and program control logic" in the UDL.
The diagram below shows the format of messages to the Up-Data Link. Each message consisted of 12 to 30 bits, depending on the message type. The first three bits, the Vehicle Address, selected which spacecraft should receive the message. (This allowed messages to be directed to the Saturn V booster, the Command Module, or the Lunar Module.3) Next, three System Address bits specified the spacecraft system to receive the message, corresponding to the four message types above. The remaining bits supplied the message text.
The contents of the message text depended on the message type. A Real Time Control (RTC) message had a six-bit value specifying the relay number as well as whether it should be turned off or on. An Apollo Guidance Computer (AGC) message had a five-bit value specifying a key on the Display/Keyboard (DSKY). For reliability, the message was encoded in 16 bits: the message, the message inverted, the message again, and a padding bit; any mismatching bits would trigger an error. A CTE message set the clock using four 6-bit values indicating seconds, minutes, hours, and days. The UDL processed the message by resetting the clock and then advancing the time by issuing the specified number of pulses to the CTE to advance the seconds, minutes, hours, and days. (This is similar to setting a digital alarm clock by advancing the digits one at a time.) Finally, the two self test messages consisted of 24-bit patterns that would exercise the UDL's internal circuitry. The results of the test were sent back to Earth via Apollo's telemetry system.
For reliability, each bit transmitted to the UDL was replaced by five "sub-bits": each "1" bit was replaced with the sub-bit sequence "01011", and each "0" bit was replaced with the complement, "10100".4 The purpose of the sub-bits was that any corrupted data would result in an invalid sub-bit code so corrupted messages could be rejected. The Up-Data Link performed this validation by matching the input data stream against "01011" or "10100". (The vehicle address at the start of a message used a different sub-bit code, ensuring that the start of the message was properly identified.) By modern standards, sub-bits are an inefficient way of providing redundancy, since the message becomes five times larger. As a consequence, the effective transmission rate was low: 200 bits per second.
There was no security in the Up-Data Link messages, apart from the need for a large transmitter. Of the systems on Apollo, only the rocket destruct system—euphemistically called the Propellant Dispersion System—was cryptographically secure.5
Since the Apollo radio system was analog, the digital sub-bits couldn't be transmitted from ground to space directly. Instead, a technique called phase-shift keying (PSK) converted the data into an audio signal. This audio signal consists of a sine wave that is inverted to indicate a 0 bit versus a 1 bit; in other words, its phase is shifted by 180 degrees for a 0 bit. The Up-Data Link box takes this audio signal as input and demodulates it to extract the digital message data. (Transmitting this audio signal from ground to the Up-Data Link required more steps that aren't relevant to the Test Set, so I'll describe them in a footnote.6)
Now that I've explained the Up-Data Link, I can describe the Test Set in more detail. The purpose of the UDL Test Set is to test the Up-Data Link system. It sends a message—as an audio signal—to the Up-Data Link box, implementing the message formatting, sub-bit encoding, and phase shift keying described above. Then it verifies the outputs from the UDL to ensure that the UDL performed the correct action.
Perhaps the most visible feature of the Test Set is the paper tape reader on the front panel: this reader is how the Test Set obtains messages to transmit. Messages are punched onto strips of paper tape, encoded as a sequence of 13 octal digits.7 After a message is read from paper tape, it is shown on the 13-digit display. The first three digits are an arbitrary message number, while the remaining 10 octal digits denote the 30-bit message to send to the UDL. Based on the type of message, specified by the System Address digit, the Test Set validates the UDL's response and indicates success or errors on the panel lights.
I created the block diagram below to explain the architecture and construction of the Test Set (click for a larger view). The system has 25 circuit boards, labeled A1 through A25;8 for the most part, they correspond to functional blocks in the diagram.
The Test Set's front panel is dominated by its display of 13 large digits. It turns out that the storage of these digits is the heart of the Test Set. This storage (A3-A9) assembles the digits as they are read from the paper tape, circulates the bits for transmission, and provides digits to the other circuits to select the message type and validate the results. To accomplish this, the 13 digit circuits are configured as a 39-bit shift register. As the message is read from the paper tape, its bits are shifted into the digit storage, right to left, and the message is shown on the display. To send the message, the shift register is reconfigured so the 10 digits form a loop, excluding the message number. As the bits cycle through the loop, the leftmost bit is encoded and transmitted. At the end of the transmission, the digits have cycled back to their original positions, so the message can be transmitted again if desired. Thus, the shift-register mechanism both deserializes the message when it is read and serializes the message for transmission.
The Test Set uses three boards (A15, A2, and A1) to expand the message with sub-bits and to encode the message into audio. The first board converts each bit into five sub-bits. The second board applies phase-shift keying (PSK) modulation, and the third board has filters to produce clean sine waves from the digital signals.
On the input side, the Test Set receives signals from the Up-Data Link (UDL) box through round military-style connectors. These input signals are buffered by boards A25, A22, A23, A10, and A24. Board 15 verifies the input sub-bits by comparing them with the transmitted sub-bits. For an AGC message, the computer signals are verified by board A14. The timing (CTE) signals are verified by boards A20 and A21. The UDL status (validity) signals are processed by board A12. Board A11 implements a switching power supply to power the interface boards.
You can see from the block diagram that the Test Set is complex and implements multiple functions. On the other hand, the block diagram also shows that it takes a lot of 1960s circuitry to implement anything. For instance, one board can only handle two digits, so the digit display alone requires seven boards. Another example is the inputs, requiring a full board for two or three input bits.
The box is built from modules that are somewhat like integrated circuits but contain discrete components. Modules like these were used in the early 1960s before ICs caught on. Each module implements a simple function such as a flip-flop or buffer. They were more convenient than individual components, since a module provided a ready-made function. They were also compact, since the components were tightly packaged inside the module.
Physically, each module has 13 pins: a row of 7 on one side and a row of 6 offset on the other side. This arrangement ensures that a module cannot be plugged in backward.
Reverse engineering these modules was difficult since they were encapsulated in plastic and the components were inaccessible. The text printed on each module hinted at its function. For example, the J-K flip-flop module above is labeled "LP FF". The "2/2G & 2/1G" module turned out to contain two NAND gates and two inverters (the 2G and 1G gates). A "2P/3G" module contains two pull-up resistors and two three-input NAND gates. Other modules provided special-purpose analog functions for the PSK modulation.
I reverse-engineered the functions of the modules by applying signals and observing the results. Conveniently, the pins are on 0.200" spacing so I could plug modules into a standard breadboard. The functions of the logic modules were generally straightforward to determine. The analog modules were more difficult; for instance, the "-3.9V" module contains a -3.9-volt Zener diode, six resistors, and three capacitors in complicated arrangements.
To determine how the modules are constructed internally, we had a module X-rayed by John McMaster and another module X-rayed in three dimensions by Lumafield. The X-rays revealed that modules were built with "cordwood construction", a common technique in the 1960s. That is, cylindrical components were mounted between two boards, stacked parallel similar to a pile of wood logs. Instead of using printed-circuit boards, the leads of the components were welded to metal strips to provide the interconnections.
For more information on these modules, see my articles Reverse-engineering a 1960s cordwood flip-flop module with X-ray CT scans and X-ray reverse-engineering a hybrid module. You can interact with the scan here.
In this section, I'll describe some of the circuit boards and point out their interesting features. A typical board has up to 15 modules, arranged as five rows of three. The modules are carefully spaced so that two boards can be meshed with the components on one board fitting into the gaps on the other board. Thus, a pair of boards forms a dense block.
Each pair of boards is attached to side rails and a mounting bracket, forming a unit.8 The bracket has ejectors to remove the board unit, since the backplane connectors grip the boards tightly. Finally, each bracket is labeled with the board numbers, the test point numbers, and the Motorola logo. The complexity of this mechanical assembly suggests that Motorola had developed an integrated prototyping system around the circuit modules, prior to the Test Set.
The photo below shows a typical board, the digit driver board. At the left, a 47-pin plug provides the connection between the board and the Test Set's backplane. At the right, 15 test connections allow the board to be probed and tested while it is installed. The board itself is a two-sided printed circuit board with gold plating. Boards are powered with +6V, -6V, and ground; the two red capacitors in the lower left filter the two voltages.
The digit driver is the most common board in the system, appearing six times.9 Each board stores two octal digits in a shift register and drives two digit displays on the front panel. Since the digits are octal, each digit requires three bits of storage, implemented with three flip-flop modules connected as a shift register. If you look closely, you can spot the six flip-flop modules, labeled "LP FF".
The digits are displayed through an unusual technology: an edge-lit lightguide display.10 From a distance, it resembles a Nixie tube, but it uses 10 lightbulbs, one for each number value, with a plastic layer for each digit. Each plastic sheet has numerous dots etched in the shape of the corresponding number. One sheet is illuminated from the edge, causing the dots in the sheet to light up and display that number. In the photo below, you can see both the illuminated and the unilluminated dots. The displays take 14 volts, but the box runs at 28 volts, so a board full of resistors on the front panel drops the voltage from 28 to 14, giving off noticeable heat in the process.
For each digit position, the driver board provides eight drive signals, one for each bulb. The drivers are implemented in "LD" modules. Since each LD module contains two drive transistors controlled by 4-input AND gates, a module supports two bulbs. Thus, a driver board holds eight LD modules in total. The LD modules are also used on other boards to drive the lights on the front panel.
The Test Set contains multiple counters to count bits, sub-bits, digits, states, and so forth. While a modern design would use binary counters, the Test Set is implemented with a circuit called a ring counter that optimizes the hardware.
For instance, to count to ten, five flip-flops are arranged as a shift register so each flip-flop sends its output to the next one. However, the last flip-flop sends its inverted output to the first. The result is that the counter will proceed: 10000, 11000, 11100, 11110, 11111 as 1 bits are shifted in at the left. But after a 1 reaches the last bit, 0 bits will be shifted in at the left: 01111, 00111, 00011, 00001, and finally 0000. Thus, the counter moves through ten states.
Why not use a 4-bit binary counter and save a flip-flop? First, the binary counter requires additional logic to go from 9 back to 0. Moreover, acting on a particular binary value requires a 4-input gate to check the four bits. But a particular value of a ring counter can be detected with a smaller 2-input gate by checking the bits on either side of the 0/1 boundary. For instance, to detect a count of 3 (11100), only the two highlighted bits need to be tested. Thus, the decoding logic is much simpler for a ring counter, which is important when each gate comes in an expensive module.
Another use of the ring counter is in the sub-state generator, counting out the five states. Since this ring counter uses three flip-flops, you might expect it to count to six. However, the first flip-flop gets one of its inputs from the second flip-flop, resulting in five states: 000, 100, 110, 011, and 001, with the 111 state skipped.11 This illustrates the flexibility of ring counters to generate arbitrary numbers of states.
Digital data could not be broadcast directly to the spacecraft, so the data was turned into an audio signal using phase-shift keying (PSK). The Test Set uses two boards (A1 and A2) to produce this signal. These boards are interesting and unusual because they are analog, unlike the other boards in the Test Set.
The idea behind phase-shift keying is to change the phase of a sine wave depending on the bit (i.e., sub-bit) value. Specifically, a 2 kHz sine wave indicated a one bit, while the sine wave was inverted for a zero bit. That is, a phase shift of 180º indicated a 0 bit. But how do you tell which sine wave is original and which is flipped? The solution was to combine the information signal with a 1 kHz reference signal that indicates the start and phase of each bit. The diagram below shows how the bits 1-0-1 are encoded into the composite audio signal that is decoded by the Up-Data Link box.
The core of the PSK modulation circuit is a transformer with a split input winding. The 2 kHz sine wave is applied to the winding's center tap. One side of the winding is grounded (by the "ø DET" module) for a 0 bit, but the other side of the winding is grounded for a 1 bit. This causes the signal to go through the winding in one direction for a 1 bit and the opposite direction for a 0 bit. The transformer's output winding thus receives an inverted signal for a 0 bit, giving the 180º phase shift seen in the second waveform above. Finally, the board produces the composite audio signal by mixing in the reference signal through a potentiometer and the "SUM" module.12
Inconveniently, some key components of the Test Set were missing; probably the most valuable components were salvaged when the box was scrapped. The missing components included the power supplies and amplifiers on the back of the box, as well as parts from PSK board A1. This board had ten white wires that had been cut, going to missing components labeled MP1, R2, L1, and L2. By studying the circuitry, I determined that MP1 had been a 4-kHz oscillator that provided the master clock for the Test Set. R2 was simply a potentiometer to adjust signal levels.
But L1 and L2 were more difficult. It took a lot of reverse-engineering before we determined that L1 and L2 were resonant filters to convert the digital waveforms to the sine waves needed for the PSK output. Marc used a combination of theory and trial-and-error to determine the inductor and capacitor values that produced a clean signal. The photo above shows our substitute filters, along with a replacement oscillator.
The Test Set receives signals from the Up-Data Link box under test and verifies that these signals are correct. The Test Set has five input boards (A22 through A25) to buffer the input signals and convert them to digital levels. The input boards also provide electrical isolation between the input signals and the Test Set, avoiding problems caused by ground loops or different voltage levels.
A typical input board is A22, which receives two input signals, supplied through coaxial cables. The board buffers the signals with op-amps, and then produces a digital signal for use by the box. The op-amp outputs go into "1 SS" isolation modules that pass the signal through to the box while ensuring isolation. These modules are optocouplers, using an LED and a phototransistor to provide isolation.13 The op-amps are powered by an isolated power supply.
Each op-amp module is a Burr-Brown Model 1506 module,14 encapsulating a transistorized op-amp into a convenient 8-pin module. The module is similar to an integrated-circuit op-amp, except it has discrete components inside and is considerably larger than an integrated circuit. Burr-Brown is said to have created the first solid-state op-amp in 1957, and started making op-amp modules around 1962.
Board A24 is also an isolated input board, but uses different circuitry. It has two modules that each contain four Schmitt triggers, circuits to sharpen up a noisy input. These modules have the puzzling label "-12+6LC". Each output goes through a "1 SS" isolation module, as with the previous input boards. This board receives the 8-bit "validity" signal from the Up-Data Link.
Board A11 is interesting: instead of sealed modules, it has a large green cube with numerous wires attached. This board turned out to be a switching power supply that implements six dual-voltage power supplies. The green cube is a transformer with 14 center-tapped windings connected to 42 pins. The transformer ensures that the power supply's outputs are isolated. This allows the op-amps on the input boards to remain electrically isolated from the rest of the Test Set.
The power supply uses a design known as a Royer Converter; the two transistors drive the transformer in a push-pull configuration. The transistors are turned on alternately at high frequency, driven by a feedback winding. The transformer has multiple windings, one for each output. Each center-tapped winding uses two diodes to produce a DC output, filtered by the large capacitors. In total, the power supply has four ±7V outputs and two ±14V outputs to supply the input boards.
This switching power supply is independent from the power supplies for the rest of the Test Set. On the back of the box, we could see where power supplies and amplifiers had been removed. Determining the voltages of the missing power supplies would have been a challenge. Fortunately, the front of the box had test points with labels for the various voltages: -6, +6, and +28, so we knew what voltages were required.
The front panel reveals many of the features of the Test Set. At the top, lights indicate the success or failure of various tests. "Sub-bit agree/error" indicates if the sub-bits read back into the Test Set match the values sent. "AGC confirm/error" shows the results of an Apollo Guidance Computer message, while "CTE confirm/error" shows the results of a Central Timing Equipment message. "Verif confirm/error" indicates if the verification message from the UDL matches the expected value for a test message. At the right, lights indicate the status of the UDL: standby, active, or powered off.
In the middle, toggle switches control the UDL operation. The "Sub-bit spoil" switch causes sub-bits to be occasionally corrupted for testing purposes. "Sub-bit compare/override" enables or disables sub-bit verification. The four switches on the right control the paper tape reader. The "Program start" switch is the important one: it causes the UDL to send one message (in "Single" mode) or multiple messages (in "Serial" mode). The Test Set can stop or continue when an error occurs ("Stop on error" / "Bypass error"). Finally, "Tape advance" causes messages to be read from paper tape, while "Tape stop" causes the UDL to re-use the current message rather than loading a new one.
The UDL provides a verification code that indicates its status. The "Verification Return" knob selects the source of this verification code: the "Direct" position uses a 4-bit verification code, while "Remote" uses an 8-bit verification code.15
At the bottom, "PSK high/low" selects the output level for the PSK signal from the Test Set. (Since the amplifier was removed from our Test Set, this switch has no effect. Likewise, the "Power On / Off" switch has no effect since the power supplies were removed. We power the Test Set with an external lab supply.) In the middle, 15 test points allow access to various signals inside the Test Set. The round elapsed time indicator shows how many hours the Test Set has been running (apparently over 12 months of continuous operation).
Once I figured out the circuitry on each board, the next problem was determining how the boards were connected. The backplane consists of rows of 47-pin sockets, one for each board. Dense white wiring runs between the sockets as well as to switches, displays, and connectors. I started beeping out the connections with a multimeter, picking a wire and then trying to find the other end. Some wires were easy since I could see both ends, but many wires disappeared into a bundle. I soon realized that manually tracing the wiring was impractically slow: with 25 boards and 47 connections per board, brute-force testing of every pair of connections would require hundreds of thousands of checks.
To automate the beeping-out of connections, I built a system that I call Beep-o-matic. The idea behind Beep-o-matic is to automatically find all the connections between two motherboard slots by plugging two special boards into the slots. By energizing all the pins on the first board in sequence, a microcontroller can detect connected pins on the second board, revealing the wiring between the two slots.
This system worked better than I expected, rapidly generating a list of connections. I still had to plug the Beep-o-matic boards into each pair of slots (about 300 combinations in total), but each scan took just a few seconds, so a full scan was practical. To find the wiring to the switches and connectors, I used a variant of the process. I plugged a board into a slot and used a program to continuously monitor the pins for changes. I went through the various switch positions and applied signals to the connectors to find the associated connections.
I started reverse-engineering the Test Set out of curiosity: given an undocumented box made from mystery modules and missing key components, could we understand it? Could we at least get the paper tape reader to run and the lights to flash? It was a tricky puzzle to figure out the modules and the circuitry, but eventually we could read a paper tape and see the results on the display.
But the box turned out to be useful. Marc has amassed a large and operational collection of Apollo communications hardware. We use the UDL Test Set to generate realistic signals that we feed into Apollo's S-band communication system. We haven't transmitted these signals to the Moon, but we have transmitted signals between antennas a few feet apart, receiving them with a box called the S-band Transponder. Moreover, we have used the Test Set to control an Up-Data Link box, a CTE clock, and a simulated Apollo Guidance Computer, reading commands from the paper tape and sending them through the complete communication path. Ironically, the one thing we haven't done with the Test Set is use it to test the Up-Data Link in the way it is intended: connecting the UDL's outputs to the Test Set and checking the panel lights.
From a wider perspective, the Test Set provides a glimpse of the vast scope of the Apollo program. This complicated box was just one part of the test apparatus for one small part of Apollo's electronics. Think of the many different electronic systems in the Apollo spacecraft, and consider the enormous effort to test them all. And electronics was just a small part of Apollo alongside the engines, mechanical structures, fuel cells, and life support systems. With all this complexity, it's not surprising that the Apollo program employed 400,000 people.
For more information, the footnotes include a list of UDL documentation16 and CuriousMarc's videos17. Follow me on Bluesky (@righto.com), Mastodon (@kenshirriff@oldbytes.space), or RSS. (I've given up on Twitter.) I worked on this project with CuriousMarc, Mike Stewart, and Eric Schlapfer. Thanks to John McMaster for X-rays, thanks to Lumafield for the CT scans, and thanks to Marcel for providing the box.
Mike found a NASA document Functional Integrated System Schematics that includes "Up Data Link GSE/SC Integrated Schematic Diagram". Unfortunately, this was not very helpful since the diagram merely shows the Test Set as a rectangle with one wire in and one wire out. The remainder of the diagram (omitted) shows that the output line passes through a dozen boxes (modulators, switches, amplifiers, and so forth) and then enters the UDL onboard the Spacecraft Command Module. At least we could confirm that the Test Set was part of the functional integrated testing of the UDL.
Notably, this diagram has the Up-Data Link Confidence Test Set denoted with "2A17". If you examine the photo of the Test Set at the top of the article, you can see that the physical box has a Dymo label "2A17", confirming that this is the same box. ↩
The table below lists the functions that could be performed by sending a "realtime command" to the Up-Data Link to activate a relay. The crew could reset any of the relays except for K1-K5 (Abort Light A and Crew Alarm).
A message selected one of 32 relays and specified if the relay should be turned on or off. The relays were magnetic latching relays, so they stayed in the selected position even when de-energized. The relay control also supported "salvo reset": four commands to reset a bank of relays at once. ↩
The Saturn V booster had a system for receiving commands from the ground, closely related to the Up-Data Link, but with some differences. The Saturn V system used the same Phase-Shift Keying (PSK) and 70 kHz subcarrier as the Up-Data Link, but the frequency of the S-band signal was different for Saturn V (2101.8 MHz). (Since the Command Module and the booster use separate frequencies, the use of different addresses in the up-data messages was somewhat redundant.) Both systems used sub-bit encoding. Both systems used three bits for the vehicle address, but the remainder of the Saturn message was different, consisting of 14 bits for the decoder address, and 18 bits for message data. A typical message for the Launch Vehicle Digital Computer (LVDC) includes a 7-bit command followed by the 7 bits inverted for error detection. The command system for the Saturn V was located in the Instrument Unit, the ring containing most of the electronic systems that was mounted at the top of the rocket, below the Lunar Module. The command system is described in Astrionics System Handbook section 6.2.
The Lunar Module also had an Up-Data system, called the Digital Up-link Assembly (DUA) and built with integrated circuits. The Digital Up-link Assembly was similar to the Command Module's Up-Data Link and allowed ground stations to control the Lunar Guidance Computer. The DUA also controlled relays to arm the ascent engine. The DUA messages consisted of three vehicle address bits, three system address bits, and 16 information bits. Unlike the Command Module's UDL, the DUA includes the 70-kHz discriminator to demodulate the sub-band. The DUA also provided a redundant up-link voice path, using the data subcarrier to transmit audio. (The Command Module had a similar redundant voice path, but the demodulation was performed in the Premodulation Processor.) The DUA was based on the Digital-Command Assembly (DCA) that received up-link commands on the development vehicles. See Lunar Module Communication System and LM10 Handbook 2.7.4.2.2. ↩
Unexpectedly, we found three different sets of sub-bit codes in different documents. The Telecommunications Study Guide says that the first digit (the Vehicle Address) encodes a one bit with the sub-bits 11011; for the remaining digits, a one bit is encoded by 10101. Apollo Digital Command System says that the first digit uses 11001 and the remainder use 10001. The schematic in Apollo Digital Up-Data Link Description shows that the first digit uses 11000 and the remainder use 01011. This encoding matches our Up-Data Link and the Test Set, although the Test Set flipped the phase in the PSK signal. (In all cases, a zero bit is encoded by inverting all five sub-bits.) ↩
To provide range safety if the rocket went off course, the Saturn V booster had a destruct system. This system used detonating fuses along the RP-1 and LOX tanks to split the tanks open. As this happened, the escape tower at the top of the rocket would pull the astronauts to safety, away from the booster. The destruct system was controlled by the Digital Range Safety Command System (DRSCS), which used a cryptographic plug to prevent a malevolent actor from blowing up the rocket.
The DRSCS—used on both the Saturn and Skylab programs—received a message consisting of a 9-character "Address" word and a 2-character "Command" word. Each character was composed of two audio-frequency tones from an "alphabet" of seven tones, reminiscent of the Dual-Tone Multi-Frequency (DTMF) signals used by Touch-Tone phones. The commands could arm the destruct circuitry, shut off propellants, disperse propellants, or switch the DRSCS off.
To make this system secure, a "code plug" was carefully installed in the rocket shortly before launch. This code plug provided the "key-of-the-day" by shuffling the mapping between tone pairs and characters. With 21 characters, there were 21! (factorial) possible keys, so the chances of spoofing a message were astronomically small. Moreover, as the System Handbook writes with understatement: "Much attention has been given to preventing execution of a catastrophic command should one component fail during flight."
For details of the range safety system, see Saturn Launch Vehicle Systems Handbook, Astrionics System Handbook (schematic in section 6.3), Apollo Spacecraft & Saturn V Launch Vehicle Pyrotechnics / Explosive Devices, The Evolution of Electronic Tracking, Optical, Telemetry, and Command Systems at the Kennedy Space Center, and Saturn V Stage I (S-IC) Overview. ↩
I explained above how the Up-Data Link message was encoded into an audio signal using phase-shift keying. However, more steps were required before this signal could be transmitted over Apollo's complicated S-band radio system. Rather than using a separate communication link for each subsystem, Apollo unified most communication over a high-frequency S-band link, calling this the "Unified S-Band". Apollo had many communication streams—voice, control data, scientific data, ranging, telemetry, television—so cramming them onto a single radio link required multiple layers of modulation, like nested Russian Matryoshka dolls with a message inside.
For the Up-Data Link, the analog PSK signal was modulated onto a subcarrier using frequency modulation. It was combined with the voice signal from ground and the pseudo-random ranging signal, and the combined signal was phase-modulated at 2106.40625 MHz and transmitted to the spacecraft through an enormous dish antenna at a ground station.
Thus, the initial message was wrapped in several layers of modulation before transmission: the binary message was expanded to five times its length by the sub-bits, modulated with Phase-Shift Keying, modulated with frequency modulation, and modulated with phase modulation.
On the spacecraft, the signal went through corresponding layers of demodulation to extract the message. A box called the Unified S-band Transceiver demodulated the phase-modulated signal and sent the data and voice signals to the pre-modulation processor (PMP). The PMP split out the voice and data subcarriers and demodulated the signals with FM discriminators. It sent the data signal (now a 2-kHz audio signal) to the Up-Data Link, where a phase-shift keying demodulator produced a binary output. Finally, each group of five sub-bits was converted to a single bit, revealing the message. ↩
The Test Set uses eight-bit paper tape, but the encoding is unusual. Each character of the paper tape consists of a three-bit octal digit, the same digit inverted, and two control bits. Because of this redundancy, the Test Set could detect errors while reading the tape.
One puzzling aspect of the paper tape reader was that we got it working, but when we tilted the Test Set on its side, the reader completely stopped working. It turned out that the reader's motor was controlled by a mercury-wetted relay, a high-current relay that uses mercury for the switch. Since mercury is a liquid, the relay would only work in the proper orientation; when we tilted the box, the mercury rolled away from the contacts. ↩
This view of the Test Set from the top shows the positions of the 25 circuit boards, A1 through A25. Most of the boards are mounted in pairs, although A1, A2, and A15 are mounted singly. Because boards A1 and A11 have larger components, they have empty slots next to them; these are not missing boards. Each board unit has two ejector levers to remove it, along with two metal tabs to lock the unit into position. The 15 numbered holes allow access to the test points for each board. (I don't know the meaning of the text "CTS" on each board unit.) The thirteen digit display modules are at the bottom, with their dropping resistors at the bottom right.
There are seven driver boards: A3 through A9. Board A3 is different from the others because it implements one digit instead of two. Instead, board A3 includes validation logic for the paper tape data. ↩
Here is the datasheet for the digit displays in the Test Set: "Numerik Indicator IND-0300". In current dollars, they cost over $200 each! The cutaway diagram shows how the bent plastic sheets are stacked and illuminated.
For amazing photos that show the internal structure of the displays, see this article. Fran Blanche's video discusses a similar display. Wikipedia has a page on lightguide displays.
While restoring the Test Set, we discovered that a few of the light bulbs were burnt out. Since displaying an octal digit only uses eight of the ten bulbs, we figured that we could swap the failed bulbs with unused bulbs from "8" or "9". It turned out that we weren't the first people to think of this—many of the "unused" bulbs were burnt out. ↩
I'll give more details on the count-to-five ring counter. The first flip-flop gets its J input from the Q' output of the last flip-flop as expected, but it gets its K input from the Q output of the second flip-flop, not the last flip-flop. If you examine the states, this causes the transition from 110 to 011 (a toggle instead of a set to 111), resulting in five states instead of six. ↩
To explain the phase-shift keying circuitry in a bit more detail, board A1 produces a 4 kHz clock signal. Board A2 divides the clock, producing a 2 kHz signal and a 1 kHz signal. The 2 kHz signal is fed into the transformer to be phase-shifted. Then the 1 kHz reference signal is mixed in to form the PSK output. Resonant filters on board A1 convert the square-wave clock signals to smooth sine waves. ↩
I was surprised to find LED opto-isolators in a device from the mid-1960s. I expected that the Test Set isolator used a light bulb, but testing showed that it switches on at 550 mV (like a diode) and operated successfully at over 100 kHz, impossible with a light bulb or photoresistor. It turns out that Texas Instruments filed a patent for an LED-based opto-isolator in 1963 and turned this into a product in 1964. The "PEX 3002" used a gallium-arsenide LED and a silicon phototransistor. Strangely, TI called this product a "molecular multiplex switch/chopper". Nowadays, an opto-isolator costs pennies, but at the time, these devices were absurdly expensive: TI's device sold for $275 (almost $3000 in current dollars). For more, see The Optical Link: A New Circuit Tool, 1965. ↩
For more information on the Burr-Brown 1506 op amp module, see Burr-Brown Handbook of Operational Amplifier RC Networks. Other documents are Burr-Brown Handbook of Operational Amplifier Applications, Op-Amp History, Operational Amplifier Milestones, and an ad for the Burr-Brown 130 op amp. ↩
I'm not sure of the meaning of the Direct versus Remote verification codes. The Block I (earlier) UDL had an 8-bit code, while the Block II (flight) UDL had a 4-bit code. The Direct code presumably comes from the UDL itself, while the Remote code is perhaps supplied through telemetry? ↩
The block diagram below shows the structure of the Up-Data Link (UDL). It uses the sub-bit decoder and a 24-stage register to deserialize the message. Based on the message, the UDL triggers relays (RTC), outputs data to the Apollo Guidance Computer (called the CMC, Command Module Computer here), sends pulses to the CTE clock, or sends validity signals back to Earth.
For details of the Apollo Up-Data system, see the diagram below (click it for a very large image). This diagram is from the Command/Service Module Systems Handbook (PDF page 64); see page 80 for written specifications of the UDL.
Other important sources of information: Apollo Digital Up-Data Link Description contains schematics and a detailed description of the UDL. Telecommunication Systems Study Guide describes the earlier UDL that included a 450 MHz FM receiver. ↩
The following CuriousMarc videos describe the Up-Data Link and the Test Set, so smash that Like button and subscribe :-)
During the Apollo flights to the Moon, the astronauts observed the spacecraft's orientation on a special instrument called the FDAI (Flight Director / Attitude Indicator). This instrument showed the spacecraft's attitude—its orientation—by rotating a ball. This ball was nicknamed the "8-ball" because it was black (albeit only on one side). The instrument also acted as a flight director, using three yellow needles to indicate how the astronauts should maneuver the spacecraft. Three more pointers showed how fast the spacecraft was rotating.
Since the spacecraft rotates along three axes (roll, pitch, and yaw), the ball also rotates along three axes. It's not obvious how the ball can rotate to an arbitrary orientation while remaining attached. In this article, I look inside an FDAI from Apollo that was repurposed for a Space Shuttle simulator1 and explain how it operates. (Spoiler: the ball mechanism is firmly attached at the "equator" and rotates in two axes. What you see is two hollow shells around the ball mechanism that spin around the third axis.)
For the missions to the Moon, the Lunar Module had two FDAIs, as shown below: one on the left for the Commander (Neil Armstrong in Apollo 11) and one on the right for the Lunar Module Pilot (Buzz Aldrin in Apollo 11). With their size and central positions, the FDAIs dominate the instrument panel, a sign of their importance. (The Command Module for Apollo also had two FDAIs, but with a different design; I won't discuss them here.2)
Each Lunar Module FDAI could display inputs from multiple sources, selected by switches on the panel.3 The ball could display attitude from either the Inertial Measurement Unit or from the backup Abort Guidance System, selected by the "ATTITUDE MON" toggle switch next to either FDAI. The pitch attitude could also be supplied by an electromechanical unit called ORDEAL (Orbital Rate Display Earth And Lunar) that simulates a circular orbit. The error indications came from the Apollo Guidance Computer, the Abort Guidance System, the landing radar, or the rendezvous radar (controlled by the "RATE/ERROR MON" switches). The pitch, roll, and yaw rate displays were driven by the Rate Gyro Assembly (RGA). The rate indications were scaled by a switch below the FDAI, selecting 25°/sec or 5°/sec.
The ball inside the indicator shows rotation around three axes. I'll first explain these axes in the context of an aircraft, since the axes of a spacecraft are more arbitrary.4 The roll axis indicates the aircraft's angle if it rolls side-to-side along its axis of flight, raising one wing and lowering the other. Thus, the indicator shows the tilt of the horizon as the aircraft rolls. The pitch axis indicates the aircraft's angle if it pitches up or down, with the indicator showing the horizon moving down or up in response. Finally, the yaw axis indicates the compass direction that the aircraft is heading, changing as the aircraft turns left or right. (A typical aircraft attitude indicator omits yaw.)
I'll illustrate how the FDAI rotates the ball in three axes, using an orange as an example. Imagine pinching the horizontal axis between two fingers with your arm extended. Rotating your arm will roll the ball counter-clockwise or clockwise (red arrow). In the FDAI, this rotation is accomplished by a motor turning the frame that holds the ball. For pitch, the ball rotates forward or backward around the horizontal axis (yellow arrow). The FDAI has a motor inside the ball to produce this rotation. Yaw is a bit more difficult to envision: imagine hemisphere-shaped shells attached to the top and bottom shafts. When a motor rotates these shells (green arrow), the hemispheres will rotate, even though the ball mechanism (the orange) remains stationary.
The diagram below shows the mechanism inside the FDAI. The indicator uses three motors to move the ball. The roll motor is attached to the FDAI's frame, while the pitch and yaw motors are inside the ball. The roll motor rotates the roll gimbal through gears, causing the ball to rotate clockwise or counterclockwise. The roll gimbal is attached to the ball mechanism at two points along the "equator"; these two points define the pitch axis. Numerous wires on the roll gimbal enter the ball along the pitch axis. The roll control transformer provides position feedback, as will be explained below.
Removing the hemispherical shells reveals the mechanism inside the ball. When the roll gimbal is rotated, this mechanism rotates with it. The pitch motor causes the ball mechanism to rotate around the pitch axis. The yaw motor and control transformer are not visible in this photo; they are behind the pitch components, oriented perpendicularly. The yaw motor turns the vertical shaft, with the two hemisphere shells attached to the top and bottom of the shaft. Thus, the yaw motor rotates the ball shells around the yaw axis, while the mechanism itself remains stationary. The control transformers for pitch and yaw provide position feedback.
Why doesn't the wiring get tangled up as the ball rotates? The solution is two sets of slip rings to implement the electrical connections. The photo below shows the first slip ring assembly, which handles rotation around the roll axis. These slip rings connect the stationary part of the FDAI to the rotating roll gimbal. The vertical metal brushes are stationary; there are 23 pairs of brushes, one for each connection to the ball mechanism. Each pair of brushes contacts one metal ring on the striped shaft, maintaining contact as the shaft rotates. Inside the shaft, 23 wires connect the circular metal contacts to the roll gimbal.
A second set of slip rings inside the ball handles rotation around the pitch axis. These rings provide the electrical connection between the wiring on the roll gimbal and the ball mechanism. The yaw axis does not use slip rings since only the hemisphere shells rotate around the yaw axis; no wires are involved.
In this section, I'll explain how the FDAI is controlled by synchros and servo loops. In the 1950s and 1960s, the standard technique for transmitting a rotational signal electrically was through a synchro. Synchros were used for everything from rotating an instrument indicator in avionics to rotating the gun on a navy battleship. A synchro produces an output that depends on the shaft's rotational position, and transmits this output signal on three wires. If you connect these wires to a second synchro, you can use the first synchro to control the second one: the shaft of the second synchro will rotate to the same angle as the first shaft. Thus, synchros are a convenient way to send a control signal electrically.
The photo below shows a typical synchro, with the input shaft on the top and five wires at the bottom: two for power and three for the output.
Internally, the synchro has a rotating winding called the rotor that is driven with 400 Hz AC. Three fixed stator windings provide the three AC output signals. As the shaft rotates, the voltages of the output signals change, indicating the angle. (A synchro resembles a transformer with three variable secondary windings.) If two connected synchros have different angles, the magnetic fields create a torque that rotates the shafts into alignment.
The downside of synchros is that they don't produce a lot of torque. The solution is to use a more powerful motor, controlled by the synchro and a feedback loop called a servo loop. The servo loop drives the motor in the appropriate direction to eliminate the error between the desired position and the current position.
The diagram below shows how the servo loop is constructed from a combination of electronics and mechanical components. The goal is to rotate the output shaft to an angle that exactly matches the input angle, specified by the three synchro wires. The control transformer compares the input angle and the output shaft position, producing an error signal. The amplifier uses this error signal to drive the motor in the appropriate direction until the error signal drops to zero. To improve the dynamic response of the servo loop, the tachometer signal is used as a negative feedback voltage. The feedback slows the motor as the system gets closer to the right position, so the motor doesn't overshoot the position and oscillate. (This is sort of like a PID controller.)
A control transformer is similar to a synchro in appearance and construction, but the rotating shaft operates as an input, not the output. In a control transformer, the three stator windings receive the inputs and the rotor winding provides the error output. If the rotor angle of the synchro transmitter and control transformer are the same, the signals cancel out and there is no error voltage. But as the difference between the two shaft angles increases, the rotor winding produces an error signal. The phase of the error signal indicates the direction of the error.
In the FDAI, the motor is a special motor/tachometer, a device that was often used in avionics servo loops. This motor is more complicated than a regular electric motor. The motor is powered by 115 volts AC at 400 hertz, but this won't spin the motor on its own. The motor also has two low-voltage control windings. Energizing the control windings with the proper phase causes the motor to spin in one direction or the other. The motor/tachometer unit also contains a tachometer to measure its speed for the feedback loop. The tachometer is driven by another 115-volt AC winding and generates a low-voltage AC signal that is proportional to the motor's rotational speed.
The photo above shows a motor/tachometer with the rotor removed. The unit has many wires because of its multiple windings. The rotor has two drums. The drum on the left, with the spiral stripes, is for the motor. This drum is a "squirrel-cage rotor", which spins due to induced currents. (There are no electrical connections to the rotor; the drums interact with the windings through magnetic fields.) The drum on the right is the tachometer rotor; it induces a signal in the output winding proportional to the speed due to eddy currents. The tachometer signal is at 400 Hz like the driving signal, either in phase or 180º out of phase, depending on the direction of rotation. For more information on how a motor/tachometer works, see my teardown.
The FDAI has three servo loops—one for each axis—and each servo loop has a separate control transformer, motor, and amplifier. The photo below shows one of the three amplifier boards. The construction is unusual and somewhat chaotic, with some components stacked on top of others to save space. Some of the component leads are long and protected with clear plastic sleeves.5 The cylindrical pulse transformer in the middle has five colorful wires coming out of it. At the left are the two transistors that drive the motor's control windings, with two capacitors between them. The transistors are mounted on a heat sink that is screwed down to the case of the amplifier assembly for cooling. Each amplifier is connected to the FDAI through seven wires with pins that plug into the sockets on the right of the board.6
The function of the board is to amplify the error signal so the motor rotates in the appropriate direction. The amplifier also uses the tachometer output from the motor unit to slow the motor as the error signal decreases, preventing overshoot. The inputs to the amplifier are 400 hertz AC signals, with the magnitude indicating the amount of error or speed and the phase indicating the direction. The two outputs from the amplifier drive the two control windings of the motor, determining which direction the motor rotates.
The schematic for the amplifier board is below. 7 The two transistors on the left amplify the error and tachometer signals, driving the pulse transformer. The outputs of the pulse transformer will have opposite phases, driving the output transistors for opposite halves of the 400 Hz cycle. This activates the motor control winding, causing the motor to spin in the desired direction.8
Bill Lear, born in 1902, was a prolific inventor with over 150 patents, creating everything from the 8-track tape to the Learjet, the iconic private plane of the 1960s. He created multiple companies in the 1920s as well as inventing one of the first car radios for Motorola before starting Lear Avionics, a company that specialized in aerospace instruments.9 Lear produced innovative aircraft instruments and flight control systems such as the F-5 automatic pilot, which received a trophy as the "greatest aviation achievement in America" for 1950.
Bill Lear went on to solve an indicator problem for the Air Force: the supersonic F-102 Delta Dagger interceptor (1953) could climb at steep angles, but existing attitude indicators could not handle nearly vertical flight. Lear developed a remote two-gyro platform that drove the cockpit indicator while avoiding "gimbal lock" during vertical flight. For the experimental X-15 rocket-powered aircraft (1959), Lear improved this indicator to handle three axes: roll, pitch, and yaw.
Meanwhile, the Siegler Corporation started in 1950 to manufacture space heaters for homes. A few years later, Siegler was acquired by John Brooks, an entrepreneur who was enthusiastic about acquisitions. In 1961, Lear Avionics became his latest acquisition, and the merged company was called Lear Siegler Incorporated, often known as LSI. (Older programmers may know Lear Siegler through the ADM-3A, an inexpensive video display terminal from 1976 that housed the display and keyboard in a stylish white case.)
The X-15's attitude indicator became the basis of the indicator for the F-4 fighter plane (the ARU/11-A). Then, after "a minimum of modification", the attitude-director indicator was used in the Gemini space program. In total, Lear Siegler provided 11 instruments in the Gemini instrument panel, with the attitude director the most important. Next, Gemini's indicator was modified to become the FDAI (flight director-attitude indicator) in the Lunar Module for Apollo.10 Lear Siegler provided numerous components for the Apollo program, from a directional gyro for the Lunar Rover to the electroluminescent display for the Apollo Guidance Computer's Display/Keyboard (DSKY).
In 1974, Lear Siegler obtained a contract to develop the Attitude-Director Indicator (ADI) for the Space Shuttle, producing a dozen ADI units for the Space Shuttle. However, by this time, Lear Siegler was losing enthusiasm for low-volume space avionics. The Instrument Division president said that "the business that we were in was an engineering business and engineers love a challenge." However, manufacturing refused to deal with the special procedures required for space manufacturing, so the Shuttle units were built by the engineering department. Lear Siegler didn't bid on later Space Shuttle avionics and the Shuttle ADI became its last space product. In the early 2000s, the Space Shuttle's instruments were upgraded to a "glass cockpit" with 11 flat-panel displays known as the Multi-function Electronic Display System (MEDS). The MEDS was produced by Lear Siegler's long-term competitor, Honeywell.
Getting back to Bill Lear, he wanted to manufacture aircraft, not just aircraft instruments, so he created the Learjet, the first mass-produced business jet. The first Learjet flew in 1963, with over 3000 eventually delivered. In the early 1970s, Lear designed a steam turbine automobile engine. Rather than water, the turbine used a secret fluorinated hydrocarbon called "Learium". Lear had visions of thousands of low-pollution "Learmobiles", but the engine failed to catch on. Lear had been on the verge of bankruptcy in the 1960s; one of his VPs explained that "the great creative minds can't be bothered with withholding taxes and investment credits and all this crap". But by the time of his death in 1978, Lear had a fortune estimated at $75 million.
Looking inside our FDAI sheds more details on the evolution of Lear Siegler's attitude directors. The photo below compares the Apollo FDAI (top) to the earlier ARU/11-A used in the F-4 aircraft (bottom). While the basic mechanism and the electronic amplifiers are the same between the two indicators, there are also substantial changes.
The biggest difference between the ARU-11/A indicator and the FDAI is that the electronics for the ARU-11/A are in a separate module that was plugged into the back of the indicator, while the FDAI includes the electronics internally, with boards mounted on the instrument frame. Specifically, the ARU-11/A has a separate unit containing a multi-winding transformer, a power supply board, and three amplifier boards (one for each axis), while the FDAI contains these components internally. The amplifier boards in the ARU-11/A and the FDAI are identical, constructed from germanium transistors rather than silicon.11 The unusual 11-pin transformers are also the same. However, the power supply boards are different, probably because the boards also contain scaling resistors that vary between the units.12 The power supply boards are also different shapes to fit the available space.
The ball assemblies of the ARU/11-A and the FDAI are almost the same, with the same motor assemblies and slip ring mechanism. The gearing has minor changes. In particular, the FDAI has two plastic gears, while the ARU/11-A uses exclusively metal gears.
The ARU/11-A has a patented pitch trim feature that was mostly—but not entirely—removed from the Apollo FDAI. The motivation for this feature is that an aircraft in level flight will be pitched up a few degrees, the "angle of attack". It is desirable for the attitude indicator to show the aircraft as horizontal, so a pitch trim knob allows the angle of attack to be canceled out on the display. The problem is that if you fly your fighter plane vertically, you want the indicator to show precisely vertical flight, rather than applying the pitch trim adjustment. The solution in the ARU-11/A is a special 8-zone potentiometer on the pitch axis that will apply the pitch trim adjustment in level flight but not in vertical flight, while providing a smooth transition between the regions. This special potentiometer is mounted inside the ball of the ARU-11/A. However, this pitch trim adjustment is meaningless for a spacecraft, so it is not implemented in the Apollo or Space Shuttle instruments. Surprisingly, the shell of the potentiometer still exists in our FDAI, but without the potentiometer itself or the wiring. Perhaps it remained to preserve the balance of the ball. In the photo below, the cylindrical potentiometer shell is indicated by an arrow. Note the holes in the front of the shell; in the ARU-11/A, the potentiometer's wiring terminals protrude through these holes, but in the FDAI, the holes are covered with tape.
Finally, the mounting of the ball hemispheres is slightly different. The ARU/11-A uses four screws at the pole of each hemisphere. Our FDAI, however, uses a single screw at each pole; the screw is tightened with a Bristol Key, causing the shaft to expand and hold the hemisphere in place.
To summarize, the Apollo FDAI occupies a middle ground: while it isn't simply a repurposed ARU-11/A, neither is it a complete redesign. Instead, it preserves the old design where possible, while stripping out undesired features such as pitch trim. The separate amplifier and mechanical units of the ARU/11-A were combined to form the larger FDAI.
The FDAI that we examined is a special unit: it was originally built for Apollo but was repurposed for a Space Shuttle simulator. Our FDAI is labeled Model 4068F, which is a Lunar Module part number. Moreover, the FDAI is internally stamped with the date "Apr. 22 1968", over a year before the first Moon landing.
However, a closer look shows that several key components were modified to make the Apollo FDAI work in the Shuttle Simulator.14 The Apollo FDAI (and the Shuttle ADI) used resolvers as inputs to control the ball, while our FDAI uses synchros. (Resolvers and synchros are similar, except resolvers use sine and cosine inputs, 90° apart, on two wire pairs, while synchros use three inputs, 120° apart, on three wires.) NASA must have replaced the three resolver control transformers in the FDAI with synchro control transformers for use in the simulator.
The Apollo FDAI used electroluminescent lighting for the display, while ours uses eight small incandescent bulbs. The metal case of our FDAI has a Dymo embossed tape label "INCANDESCENT LIGHTING", alerting users to the change from Apollo's illumination. Our FDAI also contains a step-down transformer to convert the 115 VAC input into 5 VAC to power the bulbs, while the Shuttle powered its ADI illumination directly from 5 volts.
The dial of our FDAI was repainted to match the dial of the Shuttle FDAI. The Apollo FDAI had red bands on the left and right of the dial. A close examination of our dial shows that black paint was carefully applied over the red paint, but a few specks of red paint are still visible (below). Moreover, the edges of the lines and the lozenge show slight unevenness from the repainting. Second, the Apollo FDAI had the text "ROLL RATE", "PITCH RATE", and "YAW RATE" in white next to the needle scales. In our FDAI, this text has been hidden by black paint to match the Shuttle display.13 Third, the Apollo LM FDAI had a crosshair in the center of the instrument, while our FDAI has a white U-shaped indicator, the same as the Shuttle (and the Command Module's FDAI). Finally, the ball of the Apollo FDAI has red circular regions at the poles to warn of orientations that can cause gimbal lock. Our FDAI (like the Shuttle) does not have these circles. We couldn't see any evidence that these regions were repainted, so we suspect that our FDAI has Shuttle hemispheres on the ball.
Our FDAI has also been modified electrically. Small green connectors (Micro-D MDB1) have been added between the slip rings and the motors, as well as on the gimbal arm. We think these connectors were added post-Apollo, since they are attached somewhat sloppily with glue and don't look flight-worthy. Perhaps these connectors were added to make disassembly and modification easier. Moreover, our FDAI has an elapsed time indicator, also mounted with glue.
The back of our FDAI is completely different from Apollo. First, the connector's pinout is completely different. Second, each of the six indicator needles has a mechanical adjustment as well as a trimpot (details). Finally, each of the three axes has an adjustment potentiometer.
Each Space Shuttle had three ADIs (Attitude Director Indicators), which were very similar to the Apollo FDAI, despite the name change. The photo below shows the two octagonal ADIs in the forward flight deck, one on the left in front of the Commander, and one on the right in front of the Pilot. The aft flight deck station had a third ADI.15
Our FDAI appears to have been significantly modified for use in the Shuttle simulator, as described above. However, it is much closer to the Apollo FDAI than the ADI used in the Shuttle, as I'll show in this section. My hypothesis is that the simulator was built before the Shuttle's ADI was created, so the Apollo FDAI was pressed into service.
The Shuttle's ADI was much more complicated electrically than the Apollo FDAI and our FDAI, providing improved functionality.16 For instance, while the Apollo FDAI had a simple "OFF" indicator flag to show that the indicator had lost power, the Shuttle's ADI had extensive error detection. It contained voltage level monitors to check its five power supplies. (The Shuttle ADI used three DC power sources and two AC power sources, compared to the single AC supply for Apollo.) The Shuttle's ADI also monitored the ball servos to detect position errors. Finally, it received an external "Data OK" signal. If a fault was detected by any of these monitors, the "OFF" flag was deployed to indicate that the ADI could not be trusted.
The Shuttle's ADI had six needles, the same as Apollo, but the Shuttle used feedback to make the positions more accurate. Specifically, each Shuttle needle had a feedback sensor, a Linear Variable Differential Transformer (LVDT) that generates a voltage based on the needle position. The LVDT output drove a servo feedback loop to ensure that the needle was in the exact desired position. In the Apollo FDAI, on the other hand, the needle input voltage drove a galvanometer, swinging the needle proportionally, but there was no closed loop to ensure accuracy.
I assume that the Shuttle's ADI had integrated circuit electronics to implement this new functionality, considerably more modern than the germanium transistors in the Apollo FDAI. The Shuttle probably used the same mechanical structures to rotate the ball, but I can't confirm that.
The FDAI was a critical instrument in Apollo, indicating the orientation of the spacecraft in three axes. It wasn't obvious to me how the "8-ball" can rotate in three axes while still being securely connected to the instrument. The trick is that most of the mechanism rotates in two axes, while hollow hemispherical shells provide the third rotational axis.
The FDAI has an interesting evolutionary history, from the experimental X-15 rocket plane and the F-4 fighter to the Gemini, Apollo, and Space Shuttle flights. Our FDAI has an unusual position in this history: since it was modified from Apollo to function in a Space Shuttle simulator, it shows aspects of both Apollo and the Space Shuttle indicators. It would be interesting to compare the design of a Shuttle ADI to the Apollo FDAI, but I haven't been able to find interior photos of a Shuttle ADI (or of an unmodified Apollo FDAI).17
You can see a brief video of the FDAI in motion here. For more, follow me on Bluesky (@righto.com), Mastodon (@kenshirriff@oldbytes.space), or RSS. (I've given up on Twitter.) I worked on this project with CuriousMarc, Mike Stewart, and Eric Schlapfer, so expect a video at some point. Thanks to Richard for providing the FDAI. I wrote about the F-4 fighter plane's attitude indicator here.
There were many Space Shuttle simulators, so it is unclear which simulator was the source of our FDAI. The photo below shows a simulator, with one of the ADIs indicated with an arrow. Presumably, our FDAI became available when a simulator was upgraded from physical instruments to the screens of the Multi-function Electronic Display System (MEDS).
The most complex simulators were the three Shuttle Mission Simulators, one of which could dynamically move to provide motion cues. These simulators were at the simulation facility in Houston—officially the Jake Garn Mission Simulator and Training Facility—which also had a guidance and navigation simulator, a Spacelab simulator, and integration with the WETF (Weightless Environment Training Facility, an underground pool to simulate weightlessness). The simulators were controlled by a computer complex containing dozens of networked computers. The host computers were three UNIVAC 1100/92 mainframes, 36-bit computers that ran the simulation models. These were supported by seventeen Concurrent Computer Corporation 3260 and 3280 super-minicomputers that simulated tracking, telemetry, and communication. The simulators also used real Shuttle computers running the actual flight software; these were IBM AP101S General-Purpose Computers (GPC). For more information, see Introduction to Shuttle Mission Simulation.
NASA had additional Shuttle training facilities beyond the Shuttle Mission Simulator. The Full Fuselage Trainer was a mockup of the complete Shuttle orbiter (minus the wings). It included full instrument panels (including the ADIs), but did not perform simulations. The Crew Compartment Trainers could be positioned horizontally or vertically (to simulate pre-launch operations). They contained accurate flight decks with non-functional instruments. Three Single System Trainers provided simpler mockups for astronauts to learn each system, both during normal operation and during malfunctions, before using the more complex Shuttle Mission Simulator. A list of Shuttle training facilities is in Table 3.1 of Preparing for the High Frontier. Following the end of the Shuttle program, the trainers were distributed to various museums (details). ↩
The Command Module for Apollo used a completely different FDAI (flight director-attitude indicator) that was built by Honeywell. The two designs can be easily distinguished: the Honeywell FDAI is round, while the Lear Siegler FDAI is octagonal. ↩
The FDAI's signals are more complicated than I described above. Among other things, the IMU's gimbal angles use a different coordinate system from the FDAI, so an electromechanical unit called GASTA (Gimbal Angle Sequence Transformation Assembly) used resolvers and motors to convert the coordinates. The digital attitude error signals from the computer are converted to analog by the Inertial Measurement Unit's Coupling Data Unit (IMU CDU). For attitude, the IMU is selected with the PGNS (Primary Guidance and Navigation System) switch setting. See the Lunar Module Systems Handbook, Lunar Module System Handbook Rev A, and the Apollo Operations Handbook for more.
The roll, pitch, and yaw axes of the Lunar Module are not as obvious as the axes of an airplane. The diagram below defines these axes.
The amplifier is constructed on a single-sided printed circuit board. Since the components are packed tightly on the board, routing of the board was difficult. However, some of the components have long leads, protected by plastic sleeves. This provides additional flexibility for the board routing since the leads could be positioned as desired, regardless of the geometry of the component. As a result, the style of this board is very different from modern circuit boards, where components are usually arranged in an orderly pattern. ↩
In our FDAI, the amplifier boards as well as the needle actuators are connected by pins that plug into sockets. These connections don't seem suitable for flight since they could easily vibrate loose. We suspect that the pin-and-socket connections made the module easier to reconfigure in the simulator, but were not used in flyable units. In particular, in the similar aircraft instruments (ARU/11-A) that we examined, the wires to the amplifier boards were soldered. ↩
The board has a 56-volt Zener diode, but the function of the diode is unclear. The board is powered by 28 volts, not enough voltage to activate the Zener. Perhaps the diode filters high-voltage transients, but I don't see how transients could arise in that part of the circuit. (I can imagine transients when the pulse transformer switches, but the Zener isn't connected to the transformer.) ↩
In more detail, each motor's control winding is a center-tapped winding, with the center connected to 28 volts DC. The amplifier board's output transistors will ground either side of the winding during alternate half-cycles of the 400 Hz cycle. This causes the motor to spin in one direction or the other. (Usually, control winding are driven 90° out of phase with the motor power, but I'm not sure how this phase shift is applied in the FDAI.) ↩
The history of Bill Lear and Lear Siegler is based on Love him or hate him, Bill Lear was a creator and On Course to Tomorrow: A History of Lear Siegler Instrument Division’s Manned Spaceflight Systems 1958-1981. ↩
Numerous variants of the Lear Siegler FDAI were built for Apollo, as shown before. Among other things, the length of the unit ("L MAX") varied from 8 inches to 11 inches. (Our FDAI is approximately 8 inches long.)
We examined a different ARU-11/A where the amplifier boards were not quite identical: the boards had one additional capacitor and some of the PCB traces were routed slightly differently. These boards were labeled "REV C" in the PCB copper, so they may have been later boards with a slight modification. ↩
The amplifier scaling resistors were placed on the power supply board rather than the amplifier boards, which may seem strange. The advantage of this approach is that it permitted the three amplifier boards to be identical, since the components that differ between the axes were not part of the amplifier boards. This simplified the manufacture and repair of the amplifier boards. ↩
On the front panel of our FDAI, the text "ROLL RATE", "PITCH RATE", and "YAW RATE" has been painted over. However, the text is still faintly visible (reversed) on the inside of the panel, as shown below.
The diagram below shows the internals of the Apollo LM FDAI at a high level. This diagram shows several differences between the LM FDAI and the FDAI that we examined. First, the roll, pitch, and yaw inputs to the LM FDAI are resolver inputs (i.e. sin and cos), rather than the synchro inputs to our FDAI. Second, the needle signals below are modulated on an 800 Hz carrier and are demodulated inside the FDAI. Our FDAI, however, uses positive or negative voltages to drive the needle galvanometers directly. A minor difference is that the diagram below shows the Power Off Flag wired to +28V internally, while our FDAI has the flag wired to connector pins, probably so the flag could be controlled by the simulator.
The Space Shuttle instruments were replaced with color LCD screens in the MEDS (Multifunction Electronic Display System) upgrade. This upgrade is discussed in New Displays for the Space Shuttle Cockpit. The Space Shuttle Systems Handbook shows the ADIs on the forward console (pages 263-264) and the aft console (page 275). The physical ADI is compared to the MEDS ADI display in Displays and Controls, Vol. 1 page 119. ↩
The diagram below shows the internals of the Shuttle's ADI at a high level. The Shuttle's ADI is more complicated than the Apollo FDAI, even though they have the same indicator ball and needles.
Multiple photos of the exterior of the Shuttle ADI are available here, from the National Air and Space Museum. There are interior photos of Apollo FDAIs online, but they all appear to be modified for Shuttle simulators. ↩
In 1985, Intel introduced the groundbreaking 386 processor, the first 32-bit processor in the x86 architecture. To improve performance, the 386 has a 16-byte instruction prefetch queue. The purpose of the prefetch queue is to fetch instructions from memory before they are needed, so the processor usually doesn't need to wait on memory while executing instructions. Instruction prefetching takes advantage of times when the processor is "thinking" and the memory bus would otherwise be unused.
In this article, I look at the 386's prefetch queue circuitry in detail. One interesting circuit is the incrementer, which adds 1 to a pointer to step through memory. This sounds easy enough, but the incrementer uses complicated circuitry for high performance. The prefetch queue uses a large network to shift bytes around so they are properly aligned. It also has a compact circuit to extend signed 8-bit and 16-bit numbers to 32 bits. There aren't any major discoveries in this post, but if you're interested in low-level circuits and dynamic logic, keep reading.
The photo below shows the 386's shiny fingernail-sized silicon die under a microscope. Although it may look like an aerial view of a strangely-zoned city, the die photo reveals the functional blocks of the chip. The Prefetch Unit in the upper left is the relevant block. In this post, I'll discuss the prefetch queue circuitry (highlighted in red), skipping over the prefetch control circuitry to the right. The Prefetch Unit receives data from the Bus Interface Unit (upper right) that communicates with memory. The Instruction Decode Unit receives prefetched instructions from the Prefetch Unit, byte by byte, and decodes the opcodes for execution.
The left quarter of the chip consists of stripes of circuitry that appears much more orderly than the rest of the chip. This grid-like appearance arises because each functional block is constructed (for the most part) by repeating the same circuit 32 times, once for each bit, side by side. Vertical data lines run up and down, in groups of 32 bits, connecting the functional blocks. To make this work, each circuit must fit into the same width on the die; this layout constraint forces the circuit designers to develop a circuit that uses this width efficiently without exceeding the allowed width. The circuitry for the prefetch queue uses the same approach: each circuit is 66 µm wide1 and repeated 32 times. As will be seen, fitting the prefetch circuitry into this fixed width requires some layout tricks.
The purpose of the prefetch unit is to speed up performance by reading instructions from memory before they are needed, so the processor won't need to wait to get instructions from memory. Prefetching takes advantage of times when the memory bus is otherwise idle, minimizing conflict with other instructions that are reading or writing data. In the 386, prefetched instructions are stored in a 16-byte queue, consisting of four 32-bit blocks.2
The diagram below zooms in on the prefetcher and shows its main components. You can see how the same circuit (in most cases) is repeated 32 times, forming vertical bands. At the top are 32 bus lines from the Bus Interface Unit. These lines provide the connection between the datapath and external memory, via the Bus Interface Unit. These lines form a triangular pattern as the 32 horizontal lines on the right branch off and form 32 vertical lines, one for each bit. Next are the fetch pointer and the limit register, with a circuit to check if the fetch pointer has reached the limit. Note that the two low-order bits (on the right) of the incrementer and limit check circuit are missing. At the bottom of the incrementer, you can see that some bit positions have a blob of circuitry missing from others, breaking the pattern of repeated blocks. The 16-byte prefetch queue is below the incrementer. Although this memory is the heart of the prefetcher, its circuitry takes up a relatively small area.
The bottom part of the prefetcher shifts data to align it as needed. A 32-bit value can be split across two 32-bit rows of the prefetch buffer. To handle this, the prefetcher includes a data shift network to shift and align its data. This network occupies a lot of space, but there is no active circuitry here: just a grid of horizontal and vertical wires.
Finally, the sign extend circuitry converts a signed 8-bit or 16-bit value into a signed 16-bit or 32-bit value as needed. You can see that the sign extend circuitry is highly irregular, especially in the middle. A latch stores the output of the prefetch queue for use by the rest of the datapath.
If you've written x86 programs, you probably know about the processor's Instruction Pointer (EIP) that holds the address of the next instruction to execute. As a program executes, the Instruction Pointer moves from instruction to instruction. However, it turns out that the Instruction Pointer doesn't actually exist! Instead, the 386 has an "Advance Instruction Fetch Pointer", which holds the address of the next instruction to fetch into the prefetch queue. But sometimes the processor needs to know the Instruction Pointer value, for instance, to determine the return address when calling a subroutine or to compute the destination address of a relative jump. So what happens? The processor gets the Advance Instruction Fetch Pointer address from the prefetch queue circuitry and subtracts the current length of the prefetch queue. The result is the address of the next instruction to execute, the desired Instruction Pointer value.
The Advance Instruction Fetch Pointer—the address of the next instruction to prefetch—is stored in a register at the top of the prefetch queue circuitry. As instructions are prefetched, this pointer is incremented by the prefetch circuitry. (Since instructions are fetched 32 bits at a time, this pointer is incremented in steps of four and the bottom two bits are always 0.)
But what keeps the prefetcher from prefetching too far and going outside the valid memory range? The x86 architecture infamously uses segments to define valid regions of memory. A segment has a start and end address (known as the base and limit) and memory is protected by blocking accesses outside the segment. The 386 has six active segments; the relevant one is the Code Segment that holds program instructions. Thus, the limit address of the Code Segment controls when the prefetcher must stop prefetching.3 The prefetch queue contains a circuit to stop prefetching when the fetch pointer reaches the limit of the Code Segment. In this section, I'll describe that circuit.
Comparing two values may seem trivial, but the 386 uses a few tricks to make this fast. The basic idea is to use 30 XOR gates to compare the bits of the two registers. (Why 30 bits and not 32? Since 32 bits are fetched at a time, the bottom bits of the address are 00 and can be ignored.) If the two registers match, all the XOR values will be 0, but if they don't match, an XOR value will be 1. Conceptually, connecting the XORs to a 32-input OR gate will yield the desired result: 0 if all bits match and 1 if there is a mismatch. Unfortunately, building a 32-input OR gate using standard CMOS logic is impractical for electrical reasons, as well as inconveniently large to fit into the circuit. Instead, the 386 uses dynamic logic to implement a spread-out NOR gate with one transistor in each column of the prefetcher.
The schematic below shows the implementation of one bit of the equality comparison. The mechanism is that if the two registers differ, the transistor on the right is turned on, pulling the equality bus low. This circuit is replicated 30 times, comparing all the bits: if there is any mismatch, the equality bus will be pulled low, but if all bits match, the bus remains high. The three gates on the left implement XNOR; this circuit may seem overly complicated, but it is a standard way of implementing XNOR. The NOR gate at the right blocks the comparison except during clock phase 2. (The importance of this will be explained below.)
The equality bus travels horizontally through the prefetcher, pulled low if any bits don't match. But what pulls the bus high? That's the job of the dynamic circuit below. Unlike regular static gates, dynamic logic is controlled by the processor's clock signals and depends on capacitance in the circuit to hold data. The 386 is controlled by a two-phase clock signal.4 In the first clock phase, the precharge transistor below turns on, pulling the equality bus high. In the second clock phase, the XOR circuits above are enabled, pulling the equality bus low if the two registers don't match. Meanwhile, the CMOS switch turns on in clock phase 2, passing the equality bus's value to the latch. The "keeper" circuit keeps the equality bus held high unless it is explicitly pulled low, to avoid the risk of the voltage on the equality bus slowly dissipating. The keeper uses a weak transistor to keep the bus high while inactive. But if the bus is pulled low, the keeper transistor is overpowered and turns off.
This dynamic logic reduces power consumption and circuit size. Since the bus is charged and discharged during opposite clock phases, you avoid steady current through the transistors. (In contrast, an NMOS processor like the 8086 might use a pull-up on the bus. When the bus is pulled low, would you end up with current flowing through the pull-up and the pull-down transistors. This would increase power consumption, make the chip run hotter, and limit your clock speed.)
After each prefetch, the Advance Instruction Fetch Pointer must be incremented to hold the address of the next instruction to prefetch. Incrementing this pointer is the job of the incrementer. (Because each fetch is 32 bits, the pointer is incremented by 4 each time. But in the die photo, you can see a notch in the incrementer and limit check circuit where the circuitry for the bottom two bits has been omitted. Thus, the incrementer's circuitry increments its value by 1, so the pointer (with two zero bits appended) increases in steps of 4.)
Building an incrementer circuit is straightforward, for example, you can use a chain of 30 half-adders. The problem is that incrementing a 30-bit value at high speed is difficult because of the carries from one position to the next. It's similar to calculating 99999999 + 1 in decimal; you need to tediously carry the 1, carry the 1, carry the 1, and so forth, through all the digits, resulting in a slow, sequential process.
The incrementer uses a faster approach. First, it computes all the carries at high speed, almost in parallel. Then it computes each output bit in parallel from the carries—if there is a carry into a position, it toggles that bit.
Computing the carries is straightforward in concept: if there is a block of 1 bits at the end of the value, all those bits will produce carries, but carrying is stopped by the rightmost 0 bit. For instance, incrementing binary 11011 results in 11100; there are carries from the last two bits, but the zero stops the carries. A circuit to implement this was developed at the University of Manchester in England way back in 1959, and is known as the Manchester carry chain.
In the Manchester carry chain, you build a chain of switches, one for each data bit, as shown below. For a 1 bit, you close the switch, but for a 0 bit you open the switch. (The switches are implemented by transistors.) To compute the carries, you start by feeding in a carry signal at the right The signal will go through the closed switches until it hits an open switch, and then it will be blocked.5 The outputs along the chain give us the desired carry value at each position.
Since the switches in the Manchester carry chain can all be set in parallel and the carry signal blasts through the switches at high speed, this circuit rapidly computes the carries we need. The carries then flip the associated bits (in parallel), giving us the result much faster than a straightforward adder.
There are complications, of course, in the actual implementation. The carry signal in the carry chain is inverted, so a low signal propagates through the carry chain to indicate a carry. (It is faster to pull a signal low than high.) But something needs to make the line go high when necessary. As with the equality circuitry, the solution is dynamic logic. That is, the carry line is precharged high during one clock phase and then processing happens in the second clock phase, potentially pulling the line low.
The next problem is that the carry signal weakens as it passes through multiple transistors and long lengths of wire. The solution is that each segment has a circuit to amplify the signal, using a clocked inverter and an asymmetrical inverter. Importantly, this amplifier is not in the carry chain path, so it doesn't slow down the signal through the chain.
The schematic above shows the implementation of the Manchester carry chain for a typical bit. The chain itself is at the bottom, with the transistor switch as before. During clock phase 1, the precharge transistor pulls this segment of the carry chain high. During clock phase 2, the signal on the chain goes through the "clocked inverter" at the right to produce the local carry signal. If there is a carry, the next bit is flipped by the XOR gate, producing the incremented output.6 The "keeper/amplifier" is an asymmetrical inverter that produces a strong low output but a weak high output. When there is no carry, its weak output keeps the carry chain pulled high. But as soon as a carry is detected, it strongly pulls the carry chain low to boost the carry signal.
But this circuit still isn't enough for the desired performance. The incrementer uses a second carry technique in parallel: carry skip. The concept is to look at blocks of bits and allow the carry to jump over the entire block. The diagram below shows a simplified implementation of the carry skip circuit. Each block consists of 3 to 6 bits. If all the bits in a block are 1's, then the AND gate turns on the associated transistor in the carry skip line. This allows the carry skip signal to propagate (from left to right), a block at a time. When it reaches a block with a 0 bit, the corresponding transistor will be off, stopping the carry as in the Manchester carry chain. The AND gates all operate in parallel, so the transistors are rapidly turned on or off in parallel. Then, the carry skip signal passes through a small number of transistors, without going through any logic. (The carry skip signal is like an express train that skips most stations, while the Manchester carry chain is the local train to all the stations.) Like the Manchester carry chain, the implementation of carry skip needs precharge circuits on the lines, a keeper/amplifier, and clocked logic, but I'll skip the details.
One interesting feature is the layout of the large AND gates. A 6-input AND gate is a large device, difficult to fit into one cell of the incrementer. The solution is that the gate is spread out across multiple cells. Specifically, the gate uses a standard CMOS NAND gate circuit with NMOS transistors in series and PMOS transistors in parallel. Each cell has an NMOS transistor and a PMOS transistor, and the chains are connected at the end to form the desired NAND gate. (Inverting the output produces the desired AND function.) This spread-out layout technique is unusual, but keeps each bit's circuitry approximately the same size.
The incrementer circuitry was tricky to reverse engineer because of these techniques. In particular, most of the prefetcher consists of a single block of circuitry repeated 32 times, once for each bit. The incrementer, on the other hand, consists of four different blocks of circuitry, repeating in an irregular pattern. Specifically, one block starts a carry chain, a second block continues the carry chain, and a third block ends a carry chain. The block before the ending block is different (one large transistor to drive the last block), making four variants in total. This irregular pattern is visible in the earlier photo of the prefetcher.
The bottom part of the prefetcher rotates data to align it as needed. Unlike some processors, the x86 does not enforce aligned memory accesses. That is, a 32-bit value does not need to start on a 4-byte boundary in memory. As a result, a 32-bit value may be split across two 32-bit rows of the prefetch queue. Moreover, when the instruction decoder fetches one byte of an instruction, that byte may be at any position in the prefetch queue.
To deal with these problems, the prefetcher includes an alignment network that can rotate bytes to output a byte, word, or four bytes with the alignment required by the rest of the processor.
The diagram below shows part of this alignment network. Each bit exiting the prefetch queue (top) has four wires, for rotates of 24, 16, 8, or 0 bits. Each rotate wire is connected to one of the 32 horizontal bit lines. Finally, each horizontal bit line has an output tap, going to the datapath below. (The vertical lines are in the chip's lower M1 metal layer, while the horizontal lines are in the upper M2 metal layer. For this photo, I removed the M2 layer to show the underlying layer. Shadows of the original horizontal lines are still visible.)
The idea is that by selecting one set of vertical rotate lines, the 32-bit output from the prefetch queue will be rotated left by that amount. For instance, to rotate by 8, bits are sent down the "rotate 8" lines. Bit 0 from the prefetch queue will energize horizontal line 8, bit 1 will energize horizontal line 9, and so forth, with bit 31 wrapping around to horizontal line 7. Since horizontal bit line 8 is connected to output 8, the result is that bit 0 is output as bit 8, bit 1 is output as bit 9, and so forth.
For the alignment process, one 32-bit output may be split across two 32-bit entries in the prefetch queue in four different ways, as shown above. These combinations are implemented by multiplexers and drivers. Two 32-bit multiplexers select the two relevant rows in the prefetch queue (blue and green above). Four 32-bit drivers are connected to the four sets of vertical lines, with one set of drivers activated to produce the desired shift. Each byte of each driver is wired to achieve the alignment shown above. For instance, the rotate-8 driver gets its top byte from the "green" multiplexer and the other three bytes from the "blue" multiplexer. The result is that the four bytes, split across two queue rows, are rotated to form an aligned 32-bit value.
The final circuit is sign extension. Suppose you want to add an 8-bit value to a 32-bit value. An unsigned 8-bit value can be extended to 32 bits by simply filling the upper bits with zeroes. But for a signed value, it's trickier. For instance, -1 is the eight-bit value 0xFF, but the 32-bit value is 0xFFFFFFFF. To convert an 8-bit signed value to 32 bits, the top 24 bits must be filled in with the top bit of the original value (which indicates the sign). In other words, for a positive value, the extra bits are filled with 0, but for a negative value, the extra bits are filled with 1. This process is called sign extension.9
In the 386, a circuit at the bottom of the prefetcher performs sign extension for values in instructions. This circuit supports extending an 8-bit value to 16 bits or 32 bits, as well as extending a 16-bit value to 32 bits. This circuit will extend a value with zeros or with the sign, depending on the instruction.
The schematic below shows one bit of this sign extension circuit. It consists of a latch on the left and right, with a multiplexer in the middle. The latches are constructed with a standard 386 circuit using a CMOS switch (see footnote).7 The multiplexer selects one of three values: the bit value from the swap network, 0 for sign extension, or 1 for sign extension. The multiplexer is constructed from a CMOS switch if the bit value is selected and two transistors for the 0 or 1 values. This circuit is replicated 32 times, although the bottom byte only has the latches, not the multiplexer, as sign extension does not modify the bottom byte.
The second part of the sign extension circuitry determines if the bits should be filled with 0 or 1 and sends the control signals to the circuit above. The gates on the left determine if the sign extension bit should be a 0 or a 1. For a 16-bit sign extension, this bit comes from bit 15 of the data, while for an 8-bit sign extension, the bit comes from bit 7. The four gates on the right generate the signals to sign extend each bit, producing separate signals for the bit range 31-16 and the range 15-8.
The layout of this circuit on the die is somewhat unusual. Most of the prefetcher circuitry consists of 32 identical columns, one for each bit.8 The circuitry above is implemented once, using about 16 gates (buffers and inverters are not shown above). Despite this, the circuitry above is crammed into bit positions 17 through 7, creating irregularities in the layout. Moreover, the implementation of the circuitry in silicon is unusual compared to the rest of the 386. Most of the 386's circuitry uses the two metal layers for interconnection, minimizing the use of polysilicon wiring. However, the circuit above also uses long stretches of polysilicon to connect the gates.
The diagram above shows the irregular layout of the sign extension circuitry amid the regular datapath circuitry that is 32 bits wide. The sign extension circuitry is shown in green; this is the circuitry described at the top of this section, repeated for each bit 31-8. The circuitry for bits 15-8 has been shifted upward, perhaps to make room for the sign extension control circuitry, indicated in red. Note that the layout of the control circuitry is completely irregular, since there is one copy of the circuitry and it has no internal structure. One consequence of this layout is the wasted space to the left and right of this circuitry block, the tan regions with no circuitry except vertical metal lines passing through. At the far right, a block of circuitry to control the latches has been wedged under bit 0. Intel's designers go to great effort to minimize the size of the processor die since a smaller die saves substantial money. This layout must have been the most efficient they could manage, but I find it aesthetically displeasing compared to the regularity of the rest of the datapath.
Instructions follow a tortuous path through the 386 chip. First, the Bus Interface Unit in the upper right corner reads instructions from memory and sends them over a 32-bit bus (blue) to the prefetch unit. The prefetch unit stores the instructions in the 16-byte prefetch queue.
How is an instruction executed from the prefetch queue? It turns out that there are two distinct paths. Suppose you're executing an instruction to add 12345678 to the EAX register. The prefetch queue will hold the five bytes 05 (the opcode), 78, 56, 34, and 12. The prefetch queue provides opcodes to the decoder one byte at a time over the 8-bit bus shown in red. The bus takes the lowest 8 bits from the prefetch queue's alignment network and sends this byte to a buffer (the small square at the head of the red arrow). From there, the opcode travels to the instruction decoder.10 The instruction decoder, in turn, uses large tables (PLAs) to convert the x86 instruction into a 111-bit internal format with 19 different fields.11
The data bytes of an instruction, on the other hand, go from the prefetch queue to the ALU (Arithmetic Logic Unit) through a 32-bit data bus (orange).
Unlike the previous buses, this data bus is spread out, with one wire through each column of the datapath.
This bus extends through the entire datapath so values can also be stored into registers.
For instance,
the MOV (move) instruction can store a value from an instruction (an "immediate" value) into a register.
The 386's prefetch queue contains about 7400 transistors, more than an Intel 8080 processor. (And this is just the queue itself; I'm ignoring the prefetch control logic.) This illustrates the rapid advance of processor technology: part of one functional unit in the 386 contains more transistors than an entire 8080 processor from 11 years earlier. And this unit is less than 3% of the entire 386 processor.
Every time I look at an x86 circuit, I see the complexity required to support backward compatibility, and I gain more understanding of why RISC became popular. The prefetcher is no exception. Much of the complexity is due to the 386's support for unaligned memory accesses, requiring a byte shift network to move bytes into 32-bit alignment. Moreover, at the other end of the instruction bus is the complicated instruction decoder that decodes intricate x86 instructions. Decoding RISC instructions is much easier.
In any case, I hope you've found this look at the prefetch circuitry interesting. I plan to write more about the 386, so follow me on Bluesky (@righto.com) or RSS for updates. I've written multiple articles on the 386 previously; a good place to start might be my survey of the 368 dies.
The width of the circuitry for one bit changes a few times: while the prefetch queue and segment descriptor cache use a circuit that is 66 µm wide, the datapath circuitry is a bit tighter at 60 µm. The barrel shifter is even narrower at 54.5 µm per bit. Connecting circuits with different widths wastes space, since the wiring to connect the bits requires horizontal segments to adjust the spacing. But it also wastes space to use widths that are wider than needed. Thus, changes in the spacing are rare, where the tradeoffs make it worthwhile. ↩
The Intel 8086 processor had a six-byte prefetch queue, while the Intel 8088 (used in the original IBM PC) had a prefetch queue of just four bytes. In comparison, the 16-byte queue of the 386 seems luxurious. (Some 386 processors, however, are said to only use 12 bytes due to a bug.)
The prefetch queue assumes instructions are executed in linear order, so it doesn't help with branches or loops. If the processor encounters a branch, the prefetch queue is discarded. (In contrast, a modern cache will work even if execution jumps around.) Moreover, the prefetch queue doesn't handle self-modifying code. (It used to be common for code to change itself while executing to squeeze out extra performance.) By loading code into the prefetch queue and then modifying instructions, you could determine the size of the prefetch queue: if the old instruction was executed, it must be in the prefetch queue, but if the modified instruction was executed, it must be outside the prefetch queue. Starting with the Pentium Pro, x86 processors flush the prefetch queue if a write modifies a prefetched instruction. ↩
The prefetch unit generates "linear" addresses that must be translated to physical addresses by the paging unit (ref). ↩
I don't know which phase of the clock is phase 1 and which is phase 2, so I've assigned the numbers
arbitrarily.
The 386 creates four clock signals internally from a clock input CLK2 that runs at twice the processor's clock
speed.
The 386 generates a two-phase clock with non-overlapping phases.
That is, there is a small gap between when the first phase is high and when the second phase is high.
The 386's circuitry is controlled by the clock, with alternate blocks controlled by alternate phases.
Since the clock phases don't overlap, this ensures that logic blocks are activated in sequence, allowing the
orderly flow of data.
But because the 386 uses CMOS, it also needs active-low clocks for the PMOS transistors.
You might think that you could simply use the phase 1 clock as the active-low phase 2 clock and vice versa.
The problem is that these clock phases overlap when used as active-low; there are times when both clock signals are low.
Thus, the two clock phases must be explicitly inverted to produce the two active-low clock phases.
I described the 386's clock generation circuitry in detail in this article. ↩
The Manchester carry chain is typically used in an adder, which makes it more complicated than shown here. In particular, a new carry can be generated when two 1 bits are added. Since we're looking at an incrementer, this case can be ignored.
The Manchester carry chain was first described in Parallel addition in digital computers: a new fast ‘carry’ circuit. It was developed at the University of Manchester in 1959 and used in the Atlas supercomputer. ↩
For some reason, the incrementer uses a completely different XOR circuit from the comparator, built from a multiplexer instead of logic. In the circuit below, the two CMOS switches form a multiplexer: if the first input is 1, the top switch turns on, while if the first input is a 0, the bottom switch turns on. Thus, if the first input is a 1, the second input passes through and then is inverted to form the output. But if the first input is a 0, the second input is inverted before the switch and then is inverted again to form the output. Thus, the second input is inverted if the first input is 1, which is a description of XOR.
I don't see any clear reason why two different XOR circuits were used in different parts of the prefetcher. Perhaps the available space for the layout made a difference. Or maybe the different circuits have different timing or output current characteristics. Or it could just be the personal preference of the designers. ↩
The latch circuit is based on a CMOS switch (or transmission gate) and a weak inverter. Normally, the inverter loop holds the bit. However, if the CMOS switch is enabled, its output overpowers the signal from the weak inverter, forcing the inverter loop into the desired state.
The CMOS switch consists of an NMOS transistor and a PMOS transistor in parallel. By setting the top control input high and the bottom control input low, both transistors turn on, allowing the signal to pass through the switch. Conversely, by setting the top input low and the bottom input high, both transistors turn off, blocking the signal. CMOS switches are used extensively in the 386, to form multiplexers, create latches, and implement XOR. ↩
Most of the 386's control circuitry is to the right of the datapath, rather than awkwardly wedged into the datapath. So why is this circuit different? My hypothesis is that since the circuit needs the values of bit 15 and bit 7, it made sense to put the circuitry next to bits 15 and 7; if this control circuitry were off to the right, long wires would need to run from bits 15 and 7 to the circuitry. ↩
In case this post is getting tedious, I'll provide a lighter footnote on sign extension.
The obvious mnemonic for a sign extension instruction is SEX, but that mnemonic was too risque for Intel.
The Motorola 6809 processor (1978) used this mnemonic, as did the related
68HC12 microcontroller (1996).
However, Steve Morse, architect of the 8086, stated that the sign extension instructions on the 8086 were initially
named SEX but were renamed before release to the more conservative CBW and CWD (Convert Byte to Word and Convert Word to Double word).
The DEC PDP-11 was a bit contradictory. It has a sign extend instruction with the mnemonic SXT;
the Jargon File claims that DEC engineers almost got SEX as the
assembler mnemonic, but marketing forced the change.
On the other hand,
SEX was the official abbreviation for Sign Extend
(see PDP-11 Conventions Manual, PDP-11 Paper Tape Software Handbook) and
SEX was used in the microcode for sign extend.
RCA's CDP1802 processor (1976) may have been the first with a SEX instruction, using the mnemonic SEX for the unrelated Set X instruction.
See also this Retrocomputing Stack Exchange page. ↩
It seems inconvenient to send instructions all the way across the chip from the Bus Interface Unit to the prefetch queue and then back across to the chip to the instruction decoder, which is next to the Bus Interface Unit. But this was probably the best alternative for the layout, since you can't put everything close to everything. The 32-bit datapath circuitry is on the left, organized into 32 columns. It would be nice to put the Bus Interface Unit other there too, but there isn't room, so you end up with the wide 32-bit data bus going across the chip. Sending instruction bytes across the chip is less of an impact, since the instruction bus is just 8 bits wide. ↩
See "Performance Optimizations of the 80386", Slager, Oct 1986, in Proceedings of ICCD, pages 165-168. ↩
The groundbreaking Intel 386 processor (1985) was the first 32-bit processor in the x86 architecture. Like most processors, the 386 contains numerous registers; registers are a key part of a processor because they provide storage that is much faster than main memory. The register set of the 386 includes general-purpose registers, index registers, and segment selectors, as well as registers with special functions for memory management and operating system implementation. In this blog post, I look at the silicon die of the 386 and explain how the processor implements its main registers.
It turns out that the circuitry that implements the 386's registers is much more complicated than one would expect. For the 30 registers that I examine, instead of using a standard circuit, the 386 uses six different circuits, each one optimized for the particular characteristics of the register. For some registers, Intel squeezes register cells together to double the storage capacity. Other registers support accesses of 8, 16, or 32 bits at a time. Much of the register file is "triple-ported", allowing two registers to be read simultaneously while a value is written to a third register. Finally, I was surprised to find that registers don't store bits in order: the lower 16 bits of each register are interleaved, while the upper 16 bits are stored linearly.
The photo below shows the 386's shiny fingernail-sized silicon die under a special metallurgical microscope. I've labeled the main functional blocks. For this post, the Data Unit in the lower left quadrant of the chip is the relevant component. It consists of the 32-bit arithmetic logic unit (ALU) along with the processor's main register bank (highlighted in red at the bottom). The circuitry, called the datapath, can be viewed as the heart of the processor.
The datapath is built with a regular structure: each register or ALU functional unit is a horizontal stripe of circuitry, forming the horizontal bands visible in the image. For the most part, this circuitry consists of a carefully optimized circuit copied 32 times, once for each bit of the processor. Each circuit for one bit is exactly the same width—60 µm—so the functional blocks can be stacked together like microscopic LEGO bricks. To link these circuits, metal bus lines run vertically through the datapath in groups of 32, allowing data to flow up and down through the blocks. Meanwhile, control lines run horizontally, enabling ALU operations or register reads and writes; the irregular circuitry on the right side of the Data Unit produces the signals for these control lines, activating the appropriate control lines for each instruction.
The datapath is highly structured to maximize performance while minimizing its area on the die. Below, I'll look at how the registers are implemented according to this structure.
A processor's registers are one of the most visible features of the processor architecture. The 386 processor contains 16 registers for use by application programmers, a small number by modern standards, but large enough for the time. The diagram below shows the eight 32-bit general-purpose registers. At the top are four registers called EAX, EBX, ECX, and EDX. Although these registers are 32-bit registers, they can also be treated as 16 or 8-bit registers for backward compatibility with earlier processors. For instance, the lower half of EAX can be accessed as the 16-bit register AX, while the bottom byte of EAX can be accessed as the 8-bit register AL. Moreover, bits 15-8 can also be accessed as an 8-bit register called AH. In other words, there are four different ways to access the EAX register, and similarly for the other three registers. As will be seen, these features complicate the implementation of the register set.
The bottom half of the diagram shows that the 32-bit EBP, ESI, EDI, and ESP registers can also be treated as 16-bit registers BP, SI, DI, and SP. Unlike the previous registers, these ones cannot be treated as 8-bit registers. The 386 also has six segment registers that define the start of memory segments; these are 16-bit registers. The 16 application registers are rounded out by the status flags and instruction pointer (EIP); they are viewed as 32-bit registers, but their implementation is more complicated. The 386 also has numerous registers for operating system programming, but I won't discuss them here, since they are likely in other parts of the chip.1 Finally, the 386 has numerous temporary registers that are not visible to the programmer but are used by the microcode to perform complex instructions.
The 386's registers are implemented with static RAM cells, a circuit that can hold one bit. These cells are arranged into a grid to provide multiple registers. Static RAM can be contrasted with the dynamic RAM that computers use for their main memory: dynamic RAM holds each bit in a tiny capacitor, while static RAM uses a faster but larger and more complicated circuit. Since main memory holds gigabytes of data, it uses dynamic RAM to provide dense and inexpensive storage. But the tradeoffs are different for registers: the storage capacity is small, but speed is of the essence. Thus, registers use the static RAM circuit that I'll explain below.
The concept behind a static RAM cell is to connect two inverters into a loop. If an inverter has a "0" as input, it will output a "1", and vice versa. Thus, the inverter loop will be stable, with one inverter on and one inverter off, and each inverter supporting the other. Depending on which inverter is on, the circuit stores a 0 or a 1, as shown below. Thus, the pair of inverters provides one bit of memory.
To be useful, however, the inverter loop needs a way to store a bit into it, as well as a way to read out the stored bit. To write a new value into the circuit, two signals are fed in, forcing the inverters to the desired new values. One inverter receives the new bit value, while the other inverter receives the complemented bit value. This may seem like a brute-force way to update the bit, but it works. The trick is that the inverters in the cell are small and weak, while the input signals are higher current, able to overpower the inverters.2 These signals are fed in through wiring called "bitlines"; the bitlines can also be used to read the value stored in the cell.
To control access to the register, the bitlines are connected to the inverters through pass transistors, which act as switches to control access to the inverter loop.3 When the pass transistors are on, the signals on the write lines can pass through to the inverters. But when the pass transistors are off, the inverters are isolated from the write lines. The pass transistors are turned on by a control signal, called a "wordline" since it controls access to a word of storage in the register. Since each inverter is constructed from two transistors, the circuit above consists of six transistors—thus this circuit is called a "6T" cell.
The 6T cell uses the same bitlines for reading and writing, so you can't read and write to registers simultaneously. But adding two transistors creates an "8T" circuit that lets you read from one register and write to another register at the same time. (In technical terms, the register file is two-ported.) In the 8T schematic below, the two additional transistors (G and H) are used for reading. Transistor G buffers the cell's value; it turns on if the inverter output is high, pulling the read output bitline low.4 Transistor H is a pass transistor that blocks this signal until a read is performed on this register; it is controlled by a read wordline. Note that there are two bitlines for writing (as before) along with one bitline for reading.
To construct registers (or memory), a grid is constructed from these cells. Each row corresponds to a register, while each column corresponds to a bit position. The horizontal lines are the wordlines, selecting which word to access, while the vertical lines are the bitlines, passing bits in or out of the registers. For a write, the vertical bitlines provide the 32 bits (along with their complements). For a read, the vertical bitlines receive the 32 bits from the register. A wordline is activated to read or write the selected register. To summarize: each row is a register, data flows vertically, and control signals flow horizontally.
The die photo below zooms in on the register circuitry in the lower left corner of the 386 processor. You can see the arrangement of storage cells into a grid, but note that the pattern changes from row to row. This circuitry implements 30 registers: 22 of the registers hold 32 bits, while the bottom ones are 16-bit registers. By studying the die, I determined that there are six different register circuits, which I've arbitrarily labeled (a) to (f). In this section, I'll describe these six types of registers.
I'll start at the bottom with the simplest circuit: eight 16-bit registers that I'm calling type (f). You can see a "notch" on the left side of the register file because these registers are half the width of the other registers (16 bits versus 32 bits). These registers are implemented with the 8T circuit described earlier, making them dual ported: one register can be read while another register is written. As described earlier, three vertical bus lines pass through each bit: one bitline for reading and two bitlines (with opposite polarity) for writing. Each register has two control lines (wordlines): one to select a register for reading and another to select a register for writing.
The photo below shows how four cells of type (f) are implemented on the chip. In this image, the chip's two metal layers have been removed along with most of the polysilicon wiring, showing the underlying silicon. The dark outlines indicate regions of doped silicon, while the stripes across the doped region correspond to transistor gates. I've labeled each transistor with a letter corresponding to the earlier schematic. Observe that the layout of the bottom half is a mirrored copy of the upper half, saving a bit of space. The left and right sides are approximately mirrored; the irregular shape allows separate read and wite wordlines to control the left and right halves without colliding.
The 386's register file and datapath are designed with 60 µm of width assigned to each bit. However, the register circuit above is unusual: the image above is 60 µm wide but there are two register cells side-by-side. That is, the circuit crams two bits in 60 µm of width, rather than one. Thus, this dense layout implements two registers per row (with interleaved bits), providing twice the density of the other register circuits.
If you're curious to know how the transistors above are connected, the schematic below shows how the physical arrangement of the transistors above corresponds to two of the 8T memory cells described earlier. Since the 386 has two overlapping layers of metal, it is very hard to interpret a die photo with the metal layers. But see my earlier article if you want these photos.
Above the type (f) registers are 10 registers of type (e), occupying five rows of cells. These registers are the same 8T implementation as before, but these registers are 32 bits wide instead of 16. Thus, the register takes up the full width of the datapath, unlike the previous registers. As before, the double-density circuit implements two registers per row. The silicon layout is identical (apart from being 32 bits wide instead of 16), so I'm not including a photo.
Above those registers are four (d) registers, which are more complex. They are triple-ported registers, so one register can be written while two other registers are read. (This is useful for ALU operations, for instance, since two values can be added and the result written back at the same time.) To support reading a second register, another vertical bus line is added for each bit. Each cell has two more transistors to connect the cell to the new bitline. Another wordline controls the additional read path. Since each cell has two more transistors, there are 10 transistors in total and the circuit is called 10T.
The diagram above shows four memory cells of type (d).
Each of these cells takes the full 60 µm of width, unlike the previous double-density cells.
The cells are mirrored horizontally and vertically;
this increases the density slightly since power lines can be shared between cells.
I've labeled the transistors A through H as before, as well as the two additional transistors I and J for the
second read line.
The circuit is the same as before, except for the two additional transistors, but
the silicon layout is significantly different.
Each of the (d) registers has five control lines. Two control lines select a register for reading, connecting the register to one of the two vertical read buses. The three write lines allow parts of the register to be written independently: the top 16 bits, the next 8 bits, or the bottom 8 bits. This is required by the x86 architecture, where a 32-bit register such as EAX can also be accessed as the 16-bit AX register, the 8-bit AH register, or the 8-bit AL register. Note that reading part of a register doesn't require separate control lines: the register provides all 32 bits and the reading circuit can ignore the bits it doesn't want.
Proceeding upward, the three (c) registers have a similar 10T implementation. These registers, however, do not support partial writes so all 32 bits must be written at once. As a result, these registers only require three control lines (two for reads and one for writes). With fewer control lines, the cells can be fit into less vertical space, so the layout is slightly more compact than the previous type (d) cells. The diagram below shows four type (c) rows above two type (d) rows. Although the cells have the same ten transistors, they have been shifted around somewhat.
Next are the four (b) registers, which support 16-bit writes and 32-bit writes (but not 8-bit writes). Thus, these registers have four control lines (two for reads and two for writes). The cells take slightly more vertical space than the (c) cells due to the additional control line, but the layout is almost identical.
Finally, the (a) register at the top has an unusual feature: it can receive a copy of the value in the register just below it. This value is copied directly between the registers, without using the read or write buses. This register has 3 control lines: one for read, one for write, and one for copying.
The diagram above shows a cell of type (a) above a cell of type (b). The cell of type (a) is based on the standard 8T circuit, but with six additional transistors to copy the value of the cell below. Specifically, two inverters buffer the output from cell (b), one inverter for each side of the cell. These inverters are implemented with transistors I1 through I4.5 Two transistors, S1 and S2, act as a pass-transistor switches between these inverters and the memory cell. When activated by the control line, the switch transistors allow the inverters to overwrite the memory cell with the contents of the cell below. Note that cell (a) takes considerably more vertical space because of the extra transistors.
I haven't determined the mapping between the 386's registers and the 30 physical registers, but I can speculate. First, the 386 has four registers that can be accessed as 8, 16, or 32-bit registers: EAX, EBX, ECX, and EDX. These must map onto the (d) registers, which support these access patterns.
The four index registers (ESP, EBP, ESI, and EDI) can be used as 32-bit registers or 16-bit registers, matching the four (b) registers with the same properties. Which one of these registers can be copied to the type (a) register? Maybe the stack pointer (ESP) is copied as part of interrupt handling.
The register file has eight 16-bit registers, type (f).
Since there are six 16-bit segment registers in the 386, I suspect the 16-bit registers are the segment registers and two additional registers.
The LOADALL
instruction gives some clues, suggesting that the two additional 16-bit registers are
LDT (Local Descriptor Table register) and TR (Task Register).
Moreover, LOADALL handles 10 temporary registers, matching the 10 registers of type (e) near the bottom
of the register file.
The three 32-bit registers of type (c) may be the
CR0 control register and the DR6 and DR7 debug registers.
In this article, I'm only looking at the main register file in the datapath. The 386 presumably has other registers scattered around the chip for various purposes. For instance, the Segment Descriptor Cache contains multiple registers similar to type (e), probably holding cache entries. The processor status flags and the instruction pointer (EIP) may not be implemented as discrete registers.6
To the right of the register file, a complicated block of circuitry uses seven-bit values to select registers. Two values select the registers (or constants) to read, while a third value selects the register to write. I'm currently analyzing this circuitry, which should provide more insight into how the physical registers are assigned.
There's one additional complication in the register layout. As mentioned earlier, the bottom 16 bits of the main registers can be treated as two 8-bit registers.7 For example, the 8-bit AH and AL registers form the bottom 16 bits of the EAX register. I explained earlier how the registers use multiple write control lines to allow these different parts of the register to be updated separately. However, there is also a layout problem.
To see the problem, suppose you perform an 8-bit ALU operation on the AH register, which is bits 15-8 of the EAX register. These bits must be shifted down to positions 7-0 so they can take part in the ALU operation, and then must be shifted back to positions 15-8 when stored into AH. On the other hand, if you perform an ALU operation on AL (bits 7-0 of EAX), the bits are already in position and don't need to be shifted.
To support the shifting required for 8-bit register operations, the 386's register file physically interleaves the bits of the two lower bytes (but not the high bytes). As a result, bit 0 of AL is next to bit 0 of AH in the register file, and so forth. This allows multiplexers to easily select bits from AH or AL as needed. In other words, each bit of AH and AL is in almost the correct physical position, so an 8-bit shift is not required. (If the bits were in order, each multiplexer would need to be connected to bits that are separated by eight positions, requiring inconvenient wiring.)8
The photo above shows the shuffle network. Each bit has three bus lines associated with it: two for reads and one for writes, and these all get shuffled. On the left, the lines for the 16 bits pass straight through. On the right, though, the two bytes are interleaved. This shuffle network is located below the ALU and above the register file, so data words are shuffled when stored in the register file and then unshuffled when read from the register file.9
In the photo, the lines on the left aren't quite straight. The reason is that the circuitry above is narrower than the circuitry below. For the most part, each functional block in the datapath is constructed with the same width (60 µm) for each bit. This makes the layout simpler since functional blocks can be stacked on top of each other and the vertical bus wiring can pass straight through. However, the circuitry above the registers (for the barrel shifter) is about 10% narrower (54.5 µm), so the wiring needs to squeeze in and then expand back out.10 There's a tradeoff of requiring more space for this wiring versus the space saved by making the barrel shifter narrower and Intel must have considered the tradeoff worthwhile. (My hypothesis is that since the shuffle network required additional wiring to shuffle the bits, it didn't take up more space to squeeze the wiring together at the same time.)
If you look in a book on processor design, you'll find a description of how registers can be created from static memory cells. However, the 386 illustrates that the implementation in a real processor is considerably more complicated. Instead of using one circuit, Intel used six different circuits for the registers in the 386.
The 386's register circuitry also shows the curse of backward compatibility. The x86 architecture supports 8-bit register accesses for compatibility with processors dating back to 1971. This compatibility requires additional circuitry such as the shuffle network and interleaved registers. Looking at the circuitry of x86 processors makes me appreciate some of the advantages of RISC processors, which avoid much of the ad hoc circuitry of x86 processors.
If you want more information about how the 386's memory cells were implemented, I wrote a lower-level article earlier. I plan to write more about the 386, so follow me on Bluesky (@righto.com) or RSS for updates.
The 386 has multiple registers that are only relevant to operating systems programmers (see Chapter 4 of the 386 Programmer's Reference Manual). These include the Global Descriptor Table Register (GDTR), Local Descriptor Table Register (LDTR), Interrupt Descriptor Table Register (IDTR), and Task Register (TR). There are four Control Registers CR0-CR3; CR0 controls coprocessor usage, paging, and a few other things. The six Debug Registers for hardware breakpoints are named DR0-DR3, DR6, and DR7. The two Test Registers for TLB testing are named TR6 and TR7. I expect that these registers are in the 386's Segment Unit and Paging Unit, rather than part of the processing datapath. ↩
Typically the write driver circuit generates a strong low on one of the bitlines, flipping the corresponding inverter to a high output. As soon as one inverter flips, it will force the other inverter into the right state. To support this, the pullup transistors in the inverters are weaker than normal. ↩
The pass transistor passes its signal through or blocks it. In CMOS, this is usually implemented with a transmission gate with an NMOS and a PMOS transistor in parallel. The cell uses only the NMOS transistor, which is much worse at passing a high signal than a low signal. Because there is one NMOS pass transistor on each side of the inverters, one of the transistors will be passing a low signal that will flip the state. ↩
The bitline is typically precharged to a high level for a read, and then the cell pulls the line low for a 0. This is more compact than including circuitry in each cell to pull the line high. ↩
Note that buffering is needed so the (b) cell can write to the (a) cell. If the cells were connected directly, cell (a) could overwrite cell (b) as easily as cell (b) could overwrite cell (a). With the inverters in between, cell (b) won't be affected by cell (a). ↩
In the 8086, the processor status flags are not stored as a physical register, but instead consist of flip-flops scattered throughout the chip (details). The 386 probably has a similar implementation for the flags.
In the 8086, the program counter (instruction pointer) does not exist as such. Instead, the instruction prefetch circuitry has a register holding the current prefetch address. If the program counter address is required (to push a return address or to perform a relative branch, for instance), the program counter value is derived from the prefetch address. If the 386 is similar, the program counter won't have a physical register in the register file. ↩
The x86 architecture combines two 8-bit registers to form a 16-bit register for historical reasons. The TTL-based Datapoint 2200 (1971) system had 8-bit A, B, C, D, E, H, and L registers, with the H and L registers combined to form a 16-bit indexing register for memory accesses. Intel created a microprocessor version of the Datapoint 2200's architecture, called the 8008. Intel's 8080 processor extended the register pairs so BC and DE could also be used as 16-bit registers. The 8086 kept this register design, but changed the 16-bit register names to AX, BX, CX, and DX, with the 8-bit parts called AH, AL, and so forth. Thus, the unusual physical structure of the 386's register file is due to compatibility with a programmable terminal from 1971. ↩
To support 8-bit and 16-bit operations, the 8086 processor used a similar interleaving scheme with the two 8-bit halves of a register interleaved. Since the 8086 was a 16-bit processor, though, its interleaving was simpler than the 32-bit 386. Specifically, the 8086 didn't have the upper 16 bits to deal with. ↩
The 386's constant ROM is located below the shuffle network. Thus, constants are stored with the bits interleaved in order to produce the right results. (This made the ROM contents incomprehensible until I figured out the shuffling pattern, but that's a topic for another article.) ↩
The main body of the datapath (ALU, etc.) has the same 60 µm cell width as the register file. However, the datapath is slightly wider than the register file overall. The reason? The datapath has a small amount of circuitry between bits 7 and 8 and between bits 15 and 16, in order to handle 8-bit and 16-bit operations. As a result, the logical structure of the registers is visible as stripes in the physical layout of the ALU below. (These stripes are also visible in the die photo at the beginning of this article.)
mult3
In 1977, Commodore released the PET computer, a quirky home computer that combined the processor, a tiny keyboard, a cassette drive for storage, and a trapezoidal screen in a metal unit. The Commodore PET, the Apple II, and Radio Shack's TRS-80 started the home computer market with ready-to-run computers, systems that were called in retrospect the 1977 Trinity. I did much of my early programming on the PET, so when someone offered me a non-working PET a few years ago, I took it for nostalgic reasons.
You'd think that a home computer would be easy to repair, but it turned out to be a challenge. The chips in early PETs are notorious for failures and, sure enough, we found multiple bad chips. Moreover, these RAM and ROM chips were special designs that are mostly unobtainable now. In this post, I'll summarize how we repaired the system, in case it helps anyone else.
When I first powered up the computer, I was greeted with a display full of random characters. This was actually reassuring since it showed that most of the computer was working: not just the monitor, but the video RAM, character ROM, system clock, and power supply were all operational.
With an oscilloscope, I examined signals on the system bus and found that the clock, address, and data lines were full of activity, so the 6502 CPU seemed to be operating. However, some of the data lines had three voltage levels, as shown below. This was clearly not good, and suggested that a chip on the bus was messing up the data signals.
Some helpful sites online7 suggested that if a PET gets stuck before clearing the screen, the most likely cause is a failure of a system ROM chip. Fortunately, Marc has a Retro Chip Tester, a cool device designed to test vintage ICs: not just 7400-series logic, but vintage RAMs and ROMs. Moreover, the tester knows the correct ROM contents for a ton of old computers, so it can tell if a PET ROM has the right contents.
The Retro Chip Tester showed that two of the PET's seven ROM chips had failed. These chips are MOS Technologies MPS6540, a 2K×8 ROM with a weird design that is incompatible with standard ROMs. Fortunately, several people make adapter boards that let you substitute a standard 2716 EPROM, so I ordered two adapter boards, assembled them, and Marc programmed the 2716 EPROMs from online data files. The 2716 EPROM requires a bit more voltage to program than Marc's programmer supported, but the chips seemed to have the right contents (foreshadowing).
The PET's case swings open with an arm at the left to hold it open like a car hood. The first two rows of chips at the front of the motherboard are the RAM chips. Behind the RAM are the seven ROM chips; two have been replaced by the ROM adapter boards. The 6502 processor is the large black chip behind the ROMs, toward the right.
With the adapter boards in place, I powered on the PET with great expectations of success, but it failed in precisely the same way as before, failing to clear the garbage off the screen. Marc decided it was time to use his Agilent 1670G logic analyzer to find out what was going on; (Dating back to 1999, this logic analyzer is modern by Marc's standards.) He wired up the logic analyzer to the 6502 chip, as shown below, so we could track the address bus, data bus, and the read/write signal. Meanwhile, I disassembled the ROM contents using Ghidra, so I could interpret the logic analyzer against the assembly code. (Ghidra is a program for reverse-engineering software that was developed by the NSA, strangely enough.)
The logic analyzer provided a trace of every memory access from the 6502 processor, showing what it was executing. Everything went well for a while after the system was turned on: the processor jumped to the reset vector location, did a bit of initialization, tested the memory, but then everything went haywire. I noticed that the memory test failed on the first byte. Then the software tried to get more storage by garbage collecting the BASIC program and variables. Since there wasn't any storage at all, this didn't go well and the system hung before reaching the code that clears the screen.
We tested the memory chips, using the Retro Chip Tester again, and found three bad chips. Like the ROM chips, the RAM chips are unusual: MOS Technology 6550 static RAM chip, 1K×4. By removing the bad chips and shuffling the good chips around, we reduced the 8K PET to a 6K PET. This time, the system booted, although there was a mysterious 2×2 checkerboard symbol near the middle of the screen (foreshadowing). I typed in a simple program to print "HELLO", but the results were very strange: four floating-point numbers, followed by a hang.
This behavior was very puzzling. I could successfully enter a program into the computer, which exercises a lot of the system code. (It's not like a terminal, where echoing text is trivial; the PET does a lot of processing behind the scenes to parse a BASIC program as it is entered.) However, the output of the program was completely wrong, printing floating-point numbers instead of a string.
We also encountered an intermittent problem that after turning the computer on, the boot message would be complete gibberish, as shown below. Instead of the "*** COMMODORE BASIC ***" banner, random characters and graphics would appear.
How could the computer be operating well for the most part, yet also completely wrong? We went back to the logic analyzer to find out.
I figured that the gibberish boot message would probably be the easiest thing to track down, since that happens early in the boot process. Looking at the code, I discovered that after the software tests the memory, it converts the memory size to an ASCII string using a moderately complicated algorithm.1 Then it writes the system boot message and the memory size to the screen.
The PET uses a subroutine to write text to the screen. A pointer to the text message is held in memory locations 0071 and 0072. The assembly code below stores the pointer (in the X and Y registers) into these memory locations. (This Ghidra output shows the address, the instruction bytes, and the symbolic assembler instructions.)
d5ae 86 71 STX 71 d5b0 84 72 STY 72 d5b2 60 RTS
For the code above, you'd expect the processor to read the instruction bytes 86 and 71, and then write to address 0071. Next it should read the bytes 84 and 72 and write to address 0072. However, the logic analyzer output below showed that something slightly different happened. The processor fetched instruction bytes 86 and 71 from addresses D5AE and D5AF, then wrote 00 to address 0071, as expected. Next, it fetched instruction bytes 84 and 72 as expected, but wrote 01 to address 007A, not 0072!
step address byte read/write'
112235 D5AE 86 1
112236 D5AF 71 1
112237 0071 00 0
112238 D5B0 84 1
112239 D5B1 72 1
112240 007A 01 0
This was a smoking gun. The processor had messed up and there was a one-bit error in the address. Maybe the 6502 processor issued a bad signal or maybe something else was causing problems on the bus. The consequence of this error was that the string pointer referenced random memory rather than the desired boot message, so random characters were written to the screen.
Next, I investigated why the screen had a mysterious checkerboard character. I wrote a program to scan the logic analyzer output to extract all the writes to screen memory. Most of the screen operations made sense—clearing the screen at startup and then writing the boot message—but I found one unexpected write to the screen. In the assembly code below, the Y register should be written to zero-page address 5e, and the X register should be written to the address 66, some locations used by the BASIC interpreter.
d3c8 84 5e STY 5e d3ca 86 66 STX 66
However, the logic analyzer output below showed a problem. The first line should fetch the opcode 84 from address d3c8, but the processor received the opcode 8c from the ROM, the instruction to write to a 16-bit address. The result was that instead of writing to a zero-page address, the 6502 fetched another byte to write to a 16-bit address. Specifically, it grabbed the STX instruction (86) and used that as part of the address, writing FF (a checkerboard character) to screen memory at 865E2 instead of to the BASIC data structure at 005E. Moreover, the STX instruction wasn't executed, since it was consumed as an address. Thus, not only did a stray character get written to the screen, but data structures in memory didn't get updated. It's not surprising that the BASIC interpreter went out of control when it tried to run the program.
step address byte read/write' 186600 D3C8 8C 1 186601 D3C9 5E 1 186602 D3CA 86 1 186603 865E FF 0
We concluded that a ROM was providing the wrong byte (8C) at address D3C8. This ROM turned out to be one of our replacements; the under-powered EPROM programmer had resulted in a flaky byte. Marc re-programmed the EPROM with a more powerful programmer. The system booted, but with much less RAM than expected. It turned out that another RAM chip had failed.
Finally, we got the PET to run. I typed in a simple program to generate an animated graphical pattern, a program I remembered from when I was about 133, and generated this output:
In retrospect, I should have tested all the RAM and ROM chips at the start, and we probably could have found the faults without the logic analyzer. However, the logic analyzer gave me an excuse to learn more about Ghidra and the PET's assembly code, so it all worked out in the end.4
In the end, the PET had 6 bad chips: two ROMs and four RAMs. The 6502 processor itself turned out to be fine.5 The photo below shows the 6 bad chips on top of the PET's tiny keyboard. On the top of each key, you can see the quirky graphical character set known as PETSCII.6 As for the title, I'm counting the badly-programmed ROM as half a bad chip since the chip itself wasn't bad but it was functioning erratically.
CuriousMarc created a video of the PET restoration, if you want more:
Follow me on Bluesky (@righto.com) or RSS for updates. (I'm no longer on Twitter.) Thanks to Mike Naberezny for providing the PET. Thanks to TubeTime, Mike Stewart, and especially CuriousMarc for help with the repairs. Some useful PET troubleshooting links are in the footnotes.7
Converting a number to an ASCII string is somewhat complicated on the 6502. You can't quickly divide by 10 for the decimal conversion, since the processor doesn't have a divide instruction. Instead, the PET's conversion routine has hard-coded four-byte constants: -100000000, 10000000, -100000, 100000, -10000, 1000, -100, 10, and -1. The routine repeatedly adds the first constant (i.e. subtracting 100000000) until the result is negative. Then it repeatedly adds the second constant until the result is positive, and so forth. The number of steps gives each decimal digit (after adjustment).
The same algorithm is used with the base-60 constants: -2160000, 216000, -36000, 3600, -600, and 60.
This converts the uptime count into hours, minutes, and seconds for the TIME$ variable. (The PET's basic time count is the "jiffy",
1/60th of a second.) ↩
Technically, the address 865E is not part of screen memory, which is 1000 characters starting at address 0x8000. However, the PET's address uses some shortcuts in address decoding, so 865E ends up the same as 825e, referencing the 7th character of the 16th line. ↩
Here's the source code for my demo program, which I remembered from my teenage programming.
It simply displays blocks (black, white, or gray) with 8-fold symmetry,
writing directly to screen memory with POKE statements.
(It turns out that almost anything looks good with 8-fold symmetry.)
The cryptic heart in the first PRINT statement is the clear-screen character.
So why did I suddenly decide to restore a PET that had been sitting in my garage since 2017? Well, CNN was filming an interview with Bill Gates and they wanted background footage of the 1970s-era computers that ran the Microsoft BASIC that Bill Gates wrote. Spoiler: I didn't get my computer working in time for CNN, but Marc found some other computers.
I suspected a problem with the 6502 processor because the logic analyzer showed that the 6502 read an instruction correctly but then accessed the wrong address. Eric provided a replacement 6502 chip but swapping the processor had no effect. However, reprogramming the ROM fixed both problems. Our theory is that the signal on the bus either had a timing problem or a voltage problem, causing the logic analyzer to show the correct value but the 6502 to read the wrong value. Probably the ROM had a weakly-programmed bit, causing the ROM's output for that bit to either be at an intermediate voltage or causing the output to take too long to settle to the correct voltage. The moral is that you can't always trust the logic analyzer if there are analog faults. ↩
The PETSCII graphics characters are now in Unicode in the Symbols for Legacy Computing block. ↩
The PET troubleshooting site was very helpful. The Commodore PET's Microsoft BASIC source code is here, mostly uncommented. I mapped many of the labels in the source code to the assembly code produced by Ghidra to understand the logic analyzer traces. The ROM images are here. Schematics of the PET are here. ↩↩
Most people think of machine instructions as the fundamental steps that a computer performs. However, many processors have another layer of software underneath: microcode. With microcode, instead of building the processor's control circuitry from complex logic gates, the control logic is implemented with code known as microcode, stored in the microcode ROM. To execute a machine instruction, the computer internally executes several simpler micro-instructions, specified by the microcode. In this post, I examine the microcode ROM in the original Pentium, looking at the low-level circuitry.
The photo below shows the Pentium's thumbnail-sized silicon die under a microscope. I've labeled the main functional blocks. The microcode ROM is highlighted at the right. If you look closely, you can see that the microcode ROM consists of two rectangular banks, one above the other.
The image below shows a closeup of the two microcode ROM banks. Each bank provides 45 bits of output; together they implement a micro-instruction that is 90 bits long. Each bank consists of a grid of transistors arranged into 288 rows and 720 columns. The microcode ROM holds 4608 micro-instructions, 414,720 bits in total. At this magnification, the ROM appears featureless, but it is covered with horizontal wires, each just 1.5 µm thick.
The ROM's 90 output lines are collected into a bundle of wires between the banks, as shown above. The detail shows how six of the bits exit from the banks and join the bundle. This bundle exits the ROM to the left, travels to various parts of the chip, and controls the chip's circuitry. The output lines are in the chip's top metal layer (M3): the Pentium has three layers of metal wiring with M1 on the bottom, M2 in the middle, and M3 on top.
The Pentium has a large number of bits in its micro-instruction, 90 bits compared to 21 bits in the 8086. Presumably, the Pentium has a "horizontal" microcode architecture, where the microcode bits correspond to low-level control signals, as opposed to "vertical" microcode, where the bits are encoded into denser micro-instructions. I don't have any information on the Pentium's encoding of microcode; unlike the 8086, the Pentium's patents don't provide any clues. The 8086's microcode ROM holds 512 micro-instructions, much less than the Pentium's 4608 micro-instructions. This makes sense, given the much greater complexity of the Pentium's instruction set, including the floating-point unit on the chip.
The image below shows a closeup of the Pentium's microcode ROM. For this image, I removed the three layers of metal and the polysilicon layer to expose the chip's underlying silicon. The pattern of silicon doping is visible, showing the transistors and thus the data stored in the ROM. If you have enough time, you can extract the bits from the ROM by examining the silicon and seeing where transistors are present.
Before explaining the ROM's circuitry, I'll review how an NMOS transistor is constructed. A transistor can be considered a switch between the source and drain, controlled by the gate. The source and drain regions (green) consist of silicon doped with impurities to change its semiconductor properties, forming N+ silicon. (These regions are visible in the photo above.) The gate consists of a layer of polysilicon (red), separated from the silicon by a very thin insulating oxide layer. Whenever polysilicon crosses active silicon, a transistor is formed.
Bits are stored in the ROM through the pattern of transistors in the grid. The presence or absence of a transistor stores a 0 or 1 bit.1 The closeup below shows eight bits of the microcode ROM. There are four transistors present and four gaps where transistors are missing. Thus, this part of the ROM holds four 0 bits and four 1 bits. For the diagram below, I removed the three metal layers and the polysilicon to show the underlying silicon. I colored doped (active) silicon regions green, and drew in the horizontal polysilicon lines in red. As explained above, a transistor is created if polysilicon crosses doped silicon. Thus, the contents of the ROM are defined by the pattern of silicon regions, which creates the transistors.
The horizontal silicon lines are used as wiring to provide ground to the transistors, while the horizontal polysilicon lines select one of the rows in the ROM. The transistors in that row will turn on, pulling the associated output lines low. That is, the presence of a transistor in a row causes the output to be pulled low, while the absence of a transistor causes the output line to remain high.
The diagram below shows the silicon, polysilicon, and bottom metal (M1) layers. I removed the metal from the left to reveal the silicon and polysilicon underneath, but the pattern of vertical metal lines continues there. As shown earlier, the silicon pattern forms transistors. Each horizontal metal line has a connection to ground through a metal line (not shown). The horizontal polysilicon lines select a row. When polysilicon lines cross doped silicon, the gate of a transistor is formed. Two transistors may share the drain, as in the transistor pair on the left.
The vertical metal wires form the outputs. The circles are contacts between the metal wire and the silicon of a transistor.2 Short metal jumpers connect the polysilicon lines to the metal layer above, which will be described next.
The image below shows the upper left corner of the ROM. The yellowish metal lines are the top metal layer (M3), while the reddish metal lines are the middle metal layer (M2). The thick yellowish M3 lines distribute ground to the ROM. Underneath the horizontal M3 line, a horizontal M2 line also distributes ground. The grids of black dots are numerous contacts between the M3 line and the M2 line, providing a low-resistance connection. The M2 line, in turn, connects to vertical M1 ground lines underneath—these wide vertical lines are faintly visible. These M1 lines connect to the silicon, as shown earlier, providing ground to each transistor. This illustrates the complexity of power distribution in the Pentium: the thick top metal (M3) is the primary distribution of +5 volts and ground through the chip, but power must be passed down through M2 and M1 to reach the transistors.
The other important feature above is the horizontal metal lines, which help distribute the row-select signals. As shown earlier, horizontal polysilicon lines provide the row-select signals to the transistors. However, polysilicon is not as good a conductor as metal, so long polysilicon lines have too much resistance. The solution is to run metal lines in parallel, periodically connected to the underlying polysilicon lines and reducing the overall resistance. Since the vertical metal output lines are in the M1 layer, the horizontal row-select lines run in the M2 layer so they don't collide. Short "jumpers" in the M1 layer connect the M2 lines to the polysilicon lines.
To summarize, each ROM bank contains a grid of transistors and transistor vacancies to define the bits of the ROM. The ROM is carefully designed so the different layers—silicon, polysilicon, M1, and M2—work together to maximize the ROM's performance and density.
As the Pentium executes an instruction, it provides the address of each micro-instruction to the microcode ROM. The Pentium holds this address—the micro-address—in the Microcode Address Register (MAR). The MAR is a 13-bit register located above the microcode ROM.
The diagram below shows the Microcode Address Register above the upper ROM bank. It consists of 13 bits; each bit has multiple latches to hold the value as well as any pushed subroutine micro-addresses. Between bits 7 and 8, some buffer circuitry amplifies the control signals that go to each bit's circuitry. At the right, drivers amplify the outputs from the MAR, sending the signals to the row drivers and column-select circuitry that I will discuss below. To the left of the MAR is a 32-bit register that is apparently unrelated to the microcode ROM, although I haven't determined its function.
The outputs from the Microcode Address Register select rows and columns in the microcode ROM, as I'll explain below. Bits 12 through 7 of the MAR select a block of 8 rows, while bits 6 through 4 select a row in this block. Bits 3 through 0 select one column out of each group of 16 columns to select an output bit. Thus, the microcode address controls what word is provided by the ROM.
Several different operations can be performed on the Microcode Address Register. When executing a machine instruction, the MAR must be loaded with the address of the corresponding microcode routine. (I haven't determined how this address is generated.) As microcode is executed, the MAR is usually incremented to move to the next micro-instruction. However, the MAR can branch to a new micro-address as required. The MAR also supports microcode subroutine calls; it will push the current micro-address and jump to the new micro-address. At the end of the micro-subroutine, the micro-address is popped so execution returns to the previous location. The MAR supports three levels of subroutine calls, as it contains three registers to hold the stack of pushed micro-addresses.
The MAR receives control signals and addresses from standard-cell logic located above the MAR. Strangely, in Intel's published floorplans for the Pentium, this standard-cell logic is labeled as part of the branch prediction logic, which is above it. However, carefully tracing the signals from the standard-cell logic shows that is connected to the Microcode Address Register, not the branch predictor.
As explained above, each ROM bank has 288 rows of transistors, with polysilicon lines to select one of the rows. To the right of the ROM is circuitry that activates one of these row-select lines, based on the micro-address. Each row matches a different 9-bit address. A straightforward implementation would use a 9-input AND gate for each row, matching a particular pattern of 9 address bits or their complements.
However, this implementation would require 576 very large AND gates, so it is impractical. Instead, the Pentium uses an optimized implementation with one 6-input AND gate for each group of 8 rows. The remaining three address bits are decoded once at the top of the ROM. As a result, each row only needs one gate, detecting if its group of eight rows is selected and if the particular one of eight is selected.
The schematic above shows the circuitry for a group of eight rows, slightly simplified.3 At the top, three address bits are decoded, generating eight output lines with one active at a time. The remaining six address bits are inverted, providing the bit and its complement to the decoding circuitry. Thus, the 9 bits are converted into 20 signals that flow through the decoders, a large number of wires, but not unmanageable. Each group of eight rows has a 6-input AND gate that matches a particular 6-bit address, determined by which inputs are complemented and which are not.4 The NAND gate and inverter at the left combine the 3-bit decoding and the 6-bit decoding, activating the appropriate row.
Since there are up to 720 transistors in each row, the row-select lines need to be driven with high current. Thus, the row-select drivers use large transistors, roughly 25 times the size of a regular transistor. To fit these transistors into the same vertical spacing as the rest of the decoding circuitry, a tricky packing is used. The drivers for each group of 8 rows are packed into a 3×3 grid, except the first column has two drivers (since there are 8 drivers in the group, not 9). To avoid a gap, the drivers in the first column are larger vertically and squashed horizontally.
The schematic below shows the multiplexer circuit that selects one of 16 columns for a microcode output bit. The first stage has four 4-to-1 multiplexers. Next, another 4-to-1 multiplexer selects one of the outputs. Finally, a BiCMOS driver amplifies the output for transmission to the rest of the processor.
In more detail, the ROM and the first multiplexer are essentially NMOS circuits, rather than CMOS. Specifically, the ROM's
grid of transistors is constructed from NMOS transistors that can pull a column line low, but there are no PMOS transistors in
the grid to pull the line high (since that would double the size of the ROM).
Instead, the multiplexer includes precharge transistors to pull the lines high, presumably in the clock phase before the
ROM is read.
The capacitance of the lines will keep the line high unless it is pulled low by a transistor in the grid.
One of the four transistors in the multiplexer is activated (by control signal a, b, c, or d) to select the desired line.
The output goes to a "keeper" circuit, which keeps the output high unless it is pulled low.
The keeper uses an inverter with a weak PMOS transistor that can only provide a small pull-up current.
A stronger low input will overpower this transistor, switching the state of the keeper.
The output of this multiplexer, along with the outputs of three other multiplexers, goes to the second-stage multiplexer,5
which selects one of its four inputs, based on control signals e, f, g, and h.
The output of this multiplexer is held in a latch built from two inverters. The second latch has weak transistors so the latch
can be easily forced into the desired state.
The output from the first latch goes through a CMOS switch into a second latch, creating a flip-flop.
The output from the second latch goes to a BiCMOS driver, which drives one of the 90 microcode output lines. Most processors are built from CMOS circuitry (i.e. NMOS and PMOS transistors), but the Pentium is built from BiCMOS circuitry: bipolar transistors as well as CMOS. At the time, bipolar transistors improved performance for high-current drivers; see my article on the Pentium's BiCMOS circuitry.
The diagram below shows three bits of the microcode output. This circuitry is for the upper ROM bank; the circuitry is mirrored for the lower bank. The circuitry matches the schematic above. Each of the three blocks has 16 input lines from the ROM grid. Four 4-to-1 multiplexers reduce this to 4 lines, and the second multiplexer selects a single line. The result is latched and amplified by the output driver. (Note the large square shape of the bipolar transistors.) Next is the shift register that processes the microcode ROM outputs for testing. The shift register uses XOR logic for its feedback; unlike the rest of the circuitry, the XOR logic is irregular since only some bits are fed into XOR gates.
Why does the microcode ROM have shift registers and XOR gates? The reason is that a chip such as the Pentium is very difficult to test: if one out of 3.1 million transistors goes bad, how do you detect it? For a simple processor like the 8086, you can run through the instruction set and be fairly confident that any problem would turn up. But with a complex chip, it is almost impossible to design an instruction sequence that would test every bit of the microcode ROM, every bit of the cache, and so forth. Starting with the 386, Intel added circuitry to the processor solely to make testing easier; about 2.7% of the transistors in the 386 were for testing.
The Pentium has this testing circuitry for many ROMs and PLAs, including the division PLA that caused the infamous FDIV bug. To test a ROM inside the processor, Intel added circuitry to scan the entire ROM and checksum its contents. Specifically, a pseudo-random number generator runs through each address, while another circuit computes a checksum of the ROM output, forming a "signature" word. At the end, if the signature word has the right value, the ROM is almost certainly correct. But if there is even a single bit error, the checksum will be wrong and the chip will be rejected.
The pseudo-random numbers and the checksum are both implemented with linear feedback shift registers (LFSR), a shift register along with a few XOR gates to feed the output back to the input. For more information on testing circuitry in the 386, see Design and Test of the 80386, written by Pat Gelsinger, who became Intel's CEO years later.
You'd think that implementing a ROM would be straightforward, but the Pentium's microcode ROM is surprisingly complex due to its optimized structure and its circuitry for testing. I haven't been able to determine much about how the microcode works, except that the micro-instruction is 90 bits wide and the ROM holds 4608 micro-instructions in total. But hopefully you've found this look at the circuitry interesting.
Disclaimer: this should all be viewed as slightly speculative and there are probably some errors. I didn't want to prefix every statement with "I think that..." but you should pretend it is there. I plan to write more about the implementation of the Pentium, so follow me on Bluesky (@righto.com) or RSS for updates. Peter Bosch has done some reverse engineering of the Pentium II microcode; his information is here.
It is arbitrary if a transistor corresponds to a 0 bit or a 1 bit. A transistor will pull the output line low (i.e. a 0 bit), but the signal could be inverted before it is used. More analysis of the circuitry or ROM contents would clear this up. ↩
When looking at a ROM like this, the contact pattern seems like it should tell you the contents of the ROM. Unfortunately, this doesn't work. Since a contact can be attached to one or two transistors, the contact pattern doesn't give you enough information. You need to see the silicon to determine the transistor pattern and thus the bits. ↩
I simplified the row driver schematic. The most interesting difference is that the NAND gates are optimized to use three transistors each, instead of four transistors. The trick is that one of the NMOS transistors is essentially shared across the group of 8 drivers; an inverter drives the low side of all eight gates. The second simplification is that the 6-input AND gate is implemented with two 3-input NAND gates and a NOR gate for electrical reasons.
Also, the decoder that converts 3 bits into 8 select lines is located between the banks, at the right, not at the top of the ROM as I showed in the schematic. Likewise, the inverters for the 6 row-select bits are not at the top. Instead, there are 6 inverters and 6 buffers arranged in a column to the right of the ROM, which works better for the layout. These are BiCMOS drivers so they can provide the high-current outputs necessary for the long wires and numerous transistor gates that they must drive. ↩
The inputs to the 6-input AND gate are arranged in a binary counting pattern, selecting each row in sequence. This binary arrangment is standard for a ROM's decoder circuitry and is a good way to recognize a ROM on a die. The Pentium has 36 row decoders, rather than the 64 that you'd expect from a 6-bit input. The ROM was made to the size necessary, rather than a full power of two. In most ROMs, it's difficult to determine if the ROM is addressed bottom-to-top or top-to-bottom. However, because the microcode ROM's counting pattern is truncated, one can see that the top bank starts with 0 at the top and counts downward, while the bottom bank is reversed, starting with 0 at the bottom and counting upward. ↩
A note to anyone trying to read the ROM contents: it appears that the order of entries in a group of 16 is inconsistent, so a straightforward attempt to visually read the ROM will end up with scrambled data. That is, some of the groups are reversed. I don't see any obvious pattern in which groups are reversed.
In the diagram above, look at the contacts from the select lines, connecting the select lines to the mux transistors. The contacts on the left are the mirror image of the contacts on the right, so the columns will be accessed in the opposite order. This mirroring pattern isn't consistent, though; sometimes neighboring groups are mirrored and sometimes they aren't.
I don't know why the circuitry has this layout. Sometimes mirroring adjacent groups makes the layout more efficient, but the inconsistent mirroring argues against this. Maybe an automated layout system decided this was the best way. Or maybe Intel did this to provide a bit of obfuscation against reverse engineering. ↩
In the early 1960s, Douglas Engelbart started investigating how computers could augment human intelligence: "If, in your office, you as an intellectual worker were supplied with a computer display backed up by a computer that was alive for you all day and was instantly responsive to every action you had, how much value could you derive from that?" Engelbart developed many features of modern computing that we now take for granted: the mouse,1 hypertext, shared documents, windows, and a graphical user interface. At the 1968 Joint Computer Conference, Engelbart demonstrated these innovations in a groundbreaking presentation, now known as "The Mother of All Demos."
Engelbart's demo also featured an input device known as the keyset, but unlike his other innovations, the keyset failed to catch on. The 5-finger keyset lets you type without moving your hand, entering characters by pressing multiple keys simultaneously as a chord. Christina Englebart, his daughter, loaned one of Engelbart's keysets to me. I constructed an interface to connect the keyset to USB, so that it can be used with a modern computer. The video below shows me typing with the keyset, using the mouse buttons to select upper case and special characters.2
I wrote this blog post to describe my USB keyset interface. Along the way, however, I got sidetracked by the history of The Mother of All Demos and how it obtained that name. It turns out that Engelbart's demo isn't the first demo to be called "The Mother of All Demos".
Engelbart's work has its roots in Vannevar Bush's 1945 visionary essay, "As We May Think." Bush envisioned thinking machines, along with the "memex", a compact machine holding a library of collective knowledge with hypertext-style links: "The Encyclopedia Britannica could be reduced to the volume of a matchbox." The memex could search out information based on associative search, building up a hypertext-like trail of connections.
In the early 1960s, Engelbart was inspired by Bush's essay and set out to develop means to augment human intellect: "increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems."3 Engelbart founded the Augmentation Research Center at the Stanford Research Institute (now SRI), where he and his team created a system called NLS (oN-Line System).
In 1968, Engelbart demonstrated NLS to a crowd of two thousand people at the Fall Joint Computer Conference. Engelbart gave the demo from the stage, wearing a crisp shirt and tie and a headset microphone. Engelbart created hierarchical documents, such as the shopping list above, and moved around them with hyperlinks. He demonstrated how text could be created, moved, and edited with the keyset and mouse. Other documents included graphics, crude line drawing by today's standards but cutting-edge for the time. The computer's output was projected onto a giant screen, along with video of Engelbart.
Engelbart sat at a specially-designed Herman Miller desk6 that held the keyset, keyboard, and mouse, shown above. While Engelbart was on stage in San Francisco, the SDS 9404 computer that ran the NLS software was 30 miles to the south in Menlo Park.5
To the modern eye, the demo resembles a PowerPoint presentation over Zoom, as Engelbart collaborated with Jeff Rulifson and Bill Paxton, miles away in Menlo Park. (Just like a modern Zoom call, the remote connection started with "We're not hearing you. How about now?") Jeff Rulifson browsed the NLS code, jumping between code files with hyperlinks and expanding subroutines by clicking on them. NLS was written in custom high-level languages, which they developed with a "compiler compiler" called TREE-META. The NLS system held interactive documentation as well as tracking bugs and changes. Bill Paxton interactively drew a diagram and then demonstrated how NLS could be used as a database, retrieving information by searching on keywords. (Although Engelbart was stressed by the live demo, Paxton told me that he was "too young and inexperienced to be concerned.")
Bill English, an electrical engineer, not only built the first mouse for Engelbart but was also the hardware mastermind behind the demo. In San Francisco, the screen images were projected on a 20-foot screen by a Volkswagen-sized Eidophor projector, bouncing light off a modulated oil film. Numerous cameras, video switchers and mixers created the video image. Two leased microwave links and half a dozen antennas connected SRI in Menlo Park to the demo in San Francisco. High-speed modems send the mouse, keyset, and keyboard signals from the demo back to SRI. Bill English spent months assembling the hardware and network for the demo and then managed the demo behind the scenes, assisted by a team of about 17 people.
Another participant was the famed counterculturist Stewart Brand, known for the Whole Earth Catalog and the WELL, one of the oldest online virtual communities. Brand advised Engelbart on the presentation, as well as running a camera. He'd often point the camera at a monitor to generate swirling psychedelic feedback patterns, reminiscent of the LSD that he and Engelbart had experimented with.
The demo received press attention such as a San Francisco Chronicle article titled "Fantastic World of Tomorrow's Computer". It stated, "The most fantastic glimpse into the computer future was taking place in a windowless room on the third floor of the Civic Auditorium" where Engelbart "made a computer in Menlo Park do secretarial work for him that ten efficient secretaries couldn't do in twice the time." His goal: "We hope to help man do better what he does—perhaps by as much as 50 per cent." However, the demo received little attention in the following decades.7
Engelbart continued his work at SRI for almost a decade, but as Engelbart commented with frustration, “There was a slightly less than universal perception of our value at SRI”.8 In 1977, SRI sold the Augmentation Research Center to Tymshare, a time-sharing computing company. (Timesharing was the cloud computing of the 1970s and 1980s, where companies would use time on a centralized computer.) At Tymshare, Engelbart's system was renamed AUGMENT and marketed as an office automation service, but Engelbart himself was sidelined from development, a situation that he described as sitting in a corner and becoming invisible.
Meanwhile, Bill English and some other SRI researchers9 migrated four miles south to Xerox PARC and worked on the Xerox Alto computer. The Xerox Alto incorporated many ideas from the Augmentation Research Center including the graphical user interface, the mouse, and the keyset. The Alto's keyset was almost identical to the Engelbart keyset, as can be seen in the photo below. The Alto's keyset was most popular for the networked 3D shooter game "Maze War", with the clicking of keysets echoing through the hallways of Xerox PARC.
Xerox famously failed to commercialize the ideas from the Xerox Alto, but Steve Jobs recognized the importance of interactivity, the graphical user interface, and the mouse when he visited Xerox PARC in 1979. Steve Jobs provided the Apple Lisa and Macintosh ended up with a graphical user interface and the mouse (streamlined to one button instead of three), but he left the keyset behind.10
When McDonnell Douglas acquired Tymshare in 1984, Engelbart and his software—now called Augment—had a new home.11 In 1987, McDonnell Douglas released a text editor and outline processor for the IBM PC called MiniBASE, one of the few PC applications that supported a keyset. The functionality of MiniBASE was almost identical to Engelbart's 1968 demo, but in 1987, MiniBASE was competing against GUI-based word processors such as MacWrite and Microsoft Word, so MiniBASE had little impact. Engelbart left McDonnell Douglas in 1988, forming a research foundation called the Bootstrap Institute to continue his research independently.
The name "The Mother of All Demos" has its roots in the Gulf War. In August 1990, Iraq invaded Kuwait, leading to war between Iraq and a coalition of the United States and 41 other countries. During the months of buildup prior to active conflict, Iraq's leader, Saddam Hussein, exhorted the Iraqi people to prepare for "the mother of all battles",12 a phrase that caught the attention of the media. The battle didn't proceed as Hussein hoped: during exactly 100 hours of ground combat, the US-led coalition liberated Kuwait, pushed into Iraq, crushed the Iraqi forces, and declared a ceasefire.13 Hussein's mother of all battles became the mother of all surrenders.
The phrase "mother of all ..." became the 1990s equivalent of a meme, used as a slightly-ironic superlative. It was applied to everything from The Mother of All Traffic Jams to The Mother of All Windows Books, from The Mother of All Butter Cookies to Apple calling mobile devices The Mother of All Markets.14
In 1991, this superlative was applied to a computer demo, but it wasn't Engelbart's demo. Andy Grove, Intel's president, gave a keynote speech at Comdex 1991 entitled The Second Decade: Computer-Supported Collaboration, a live demonstration of his vision for PC-based video conferencing and wireless communication in the PC's second decade. This complex hour-long demo required almost six months to prepare, with 15 companies collaborating. Intel called this demo "The Mother of All Demos", a name repeated in the New York Times, San Francisco Chronicle, Fortune, and PC Week.15 Andy Grove's demo was a hit, with over 20,000 people requesting a video tape, but the demo was soon forgotten.
In 1994, Wired writer Steven Levy wrote Insanely Great: The Life and Times of Macintosh, the Computer that Changed Everything.8 In the second chapter of this comprehensive book, Levy explained how Vannevar Bush and Doug Engelbart "sparked a chain reaction" that led to the Macintosh. The chapter described Engelbart's 1968 demo in detail including a throwaway line saying, "It was the mother of all demos."16 Based on my research, I think this is the source of the name "The Mother of All Demos" for Engelbart's demo.
By the end of the century, multiple publications echoed Levy's catchy phrase. In February 1999, the San Jose Mercury News had a special article on Engelbart, saying that the demonstration was "still called 'the mother of all demos'", a description echoed by the industry publication Computerworld.17 The book Nerds: A Brief History of the Internet stated that the demo "has entered legend as 'the mother of all demos'". By this point, Engelbart's fame for the "mother of all demos" was cemented and the phrase became near-obligatory when writing about him. The classic Silicon Valley history Fire in the Valley (1984), for example, didn't even mention Engelbart but in the second edition (2000), "The Mother of All Demos" had its own chapter.
Getting back to the keyset interface, the keyset consists of five microswitches, triggered by the five levers. The switches are wired to a standard DB-25 connector. I used a Teensy 3.6 microcontroller board for the interface, since this board can act both as a USB device and as a USB host. As a USB device, the Teensy can emulate a standard USB keyboard. As a USB host, the Teensy can receive input from a standard USB mouse.
Connecting the keyset to the Teensy is (almost) straightforward, wiring the switches to five data inputs on the Teensy and the common line connected to ground.
The Teensy's input lines can be configured with pullup resistors inside the microcontroller. The result is that a data line shows 1 by default and
0 when the corresponding key is pressed.
One complication is that the keyset apparently has a 1.5 kΩ between the leftmost button and ground, maybe to indicate that the device is plugged in.
This resistor caused that line to always appear low to the Teensy.
To counteract this and allow the Teensy to read the pin, I connected a 1 kΩ pullup resistor to that one line.
Reading the keyset and sending characters over USB is mostly straightforward, but there are a few complications. First, it's unlikely that the user will press multiple keyset buttons at exactly the same time. Moreover, the button contacts may bounce. To deal with this, I wait until the buttons have a stable value for 100 ms (a semi-arbitrary delay) before sending a key over USB.
The second complication is that with five keys, the keyset only supports 32 characters. To obtain upper case, numbers, special characters, and control characters, the keyset is designed to be used in conjunction with mouse buttons. Thus, the interface needs to act as a USB host, so I can plug in a USB mouse to the interface. If I want the mouse to be usable as a mouse, not just buttons in conjunction with the keyset, the interface mus forward mouse events over USB. But it's not that easy, since mouse clicks in conjunction with the keyset shouldn't be forwarded. Otherwise, unwanted clicks will happen while using the keyset.
To emulate a keyboard, the code uses the Keyboard library. This library provides
an API to send characters to the destination computer.
Inconveniently, the simplest method, print(), supports only regular characters, not special characters like ENTER or BACKSPACE. For those, I needed to
use the lower-level press() and release() methods.
To read the mouse buttons,
the code uses the USBHost_t36 library, the Teensy version of the USB Host library.
Finally, to pass mouse motion through to the destination computer, I use the Mouse library.
If you want to make your own keyset, Eric Schlaepfer has a model here.
Engelbart claimed that learning a keyset wasn't difficult—a six-year-old kid could learn it in less than a week—but I'm not willing to invest much time into learning it. In my brief use of the keyset, I found it very difficult to use physically. Pressing four keys at once is difficult, with the worst being all fingers except the ring finger. Combining this with a mouse button or two at the same time gave me the feeling that I was sight-reading a difficult piano piece. Maybe it becomes easier with use, but I noticed that Alto programs tended to treat the keyset as function keys, rather than a mechanism for typing with chords.18 David Liddle of Xerox PARC said, "We found that [the keyset] was tending to slow people down, once you got away from really hot [stuff] system programmers. It wasn't quite so good if you were giving it to other engineers, let alone clerical people and so on."
If anyone else has a keyset that they want to connect via USB (unlikely as it may be), my code is on github.19 Thanks to Christina Engelbart for loaning me the keyset. Thanks to Bill Paxton for answering my questions. Follow me on Bluesky (@righto.com) or RSS for updates.
Engelbart's use of the mouse wasn't arbitrary, but based on research. In 1966, shortly after inventing the mouse, Engelbart carried out a NASA-sponsored study that evaluated six input devices: two types of joysticks, a Graphacon positioner, the mouse, a light pen, and a control operated by the knees (leaving the hands free). The mouse, knee control, and light pen performed best, with users finding the mouse satisfying to use. Although inexperienced subjects had some trouble with the mouse, experienced subjects considered it the best device.
The information sheet below from the Augmentation Research Center shows what keyset chords correspond to each character. I used this encoding for my interface software. Each column corresponds to a different combination of mouse buttons.
The special characters above are <CD> (Command Delete, i.e. cancel a partially-entered command), <BC> (Backspace Character), <OK> (confirm command), <BW>(Backspace Word), <RC> (Replace Character), <ESC> (which does filename completion).
NLS and the Augment software have the concept of a viewspec, a view specification that controls the view of a file. For instance, viewspecs can expand or collapse an outline to show more or less detail, filter the content, or show authorship of sections. The keyset can select viewspecs, as shown below.
Viewsets are explained in more detail in The Mother of All Demos. For my keyset interface, I ignored viewspecs since I don't have software to use these inputs, but it would be easy to modify the code to output the desired viewspec characters.
See Augmenting Human Intellect: A Conceptual Framework, Engelbart's 1962 report. ↩
Engelbart used an SDS 940 computer running the Berkeley Timesharing System. The computer had 64K words of core memory, with 4.5 MB of drum storage for swapping and 96 MB of disk storage for files. For displays, the computer drove twelve 5" high-resolution CRTs, but these weren't viewed directly. Instead, each CRT had a video camera pointed at it and the video was redisplayed on a larger display in a work station in each office.
The SDS 940 was a large 24-bit scientific computer, built by Scientific Data Systems. Although SDS built the first integrated-circuit-based commercial computer in 1965 (the SDS 92), the SDS 940 was a transistorized system. It consisted of multiple refrigerator-sized cabinets, as shown below. Since each memory cabinet held 16K words and the computer at SRI had 64K, SRI's computer had two additional cabinets of memory.
In the late 1960s, Xerox wanted to get into the computer industry, so Xerox bought Scientific Data Systems in 1969 for $900 million (about $8 billion in current dollars). The acquisition was a disaster. After steadily losing money, Xerox decided to exit the mainframe computer business in 1975. Xerox's CEO summed up the purchase: "With hindsight, we would not have done the same thing." ↩
The Mother of All Demos is on YouTube, as well as a five-minute summary for the impatient. ↩
The desk for the keyset and mouse was designed by Herman Miller, the office furniture company. Herman Miller worked with SRI to design the desks, chairs, and office walls as part of their plans for the office of the future. Herman Miller invented the cubicle office in 1964, creating a modern replacement for the commonly used open office arrangement. ↩
Engelbart's demo is famous now, but for many years it was ignored. For instance, Electronic Design had a long article on Engelbart's work in 1969 (putting the system on the cover), but there was no mention of the demo.
But by the 1980s, the Engelbart demo started getting attention. The 1986 documentary Silicon Valley Boomtown had a long section on Engelbart's work and the demo. By 1988, the New York Times was referring to the demo as legendary. ↩
Levy had written about Engelbart a decade earlier, in the May 1984 issue of the magazine Popular Computing. The article focused on the mouse, recently available to the public through the Apple Lisa and the IBM PC (as an option). The big issue at the time was how many buttons a mouse should have: three like Engelbart's mouse, the one button that Apple used, or two buttons as Bill Gates preferred. But Engelbart's larger vision also came through in Levy's interview along with his frustration that most of his research had been ignored, overshadowed by the mouse. Notably, there was no mention of Engelbart's 1968 demo in the article. ↩↩
The SRI researchers who moved to Xerox include Bill English, Charles Irby, Jeff Rulifson, Bill Duval, and Bill Paxton (details). ↩
In 2023, Xerox donated the entire Xerox PARC research center to SRI. The research center remained in Palo Alto but became part of SRI. In a sense, this closed the circle, since many of the people and ideas from SRI had gone to PARC in the 1970s. However, both PARC and SRI had changed radically since the 1970s, with the cutting edge of computer research moving elsewhere. ↩
For a detailed discussion of the Augment system, see Tymshare's Augment: Heralding a New Era, Oct 1978. Augment provided a "broad range of information handling capability" that was not available elsewhere. Unlike other word processing systems, Augment was targeted at the professional, not clerical workers, people who were "eager to explore the open-ended possibilities" of the interactive process.
The main complaints about Augment were its price and that it was not easy to use. Accessing Engelbart's NLS system over ARPANET cost an eye-watering $48,000 a year (over $300,000 a year in current dollars). Tymshare's Augment service was cheaper (about $80 an hour in current dollars), but still much more expensive than a standard word processing service.
Overall, the article found that Augment users were delighted with the system: "It is stimulating to belong to the electronic intelligentsia." Users found it to be "a way of life—an absorbing, enriching experience". ↩
William Safire provided background in the New York Times, explaining that "the mother of all battles" originally referred to the battle of Qadisiya in A.D. 636, and Saddam Hussein was referencing that ancient battle. A translator responded, however, that the Arabic expression would be better translated as "the great battle" than "the mother of all battles." ↩
The end of the Gulf War left Saddam Hussein in control of Iraq and left thousands of US troops in Saudi Arabia. These factors would turn out to be catastrophic in the following years. ↩
At the Mobile '92 conference, Apple's CEO, John Sculley, said personal communicators could be "the mother of all markets," while Andy Grove of Intel said that the idea of a wireless personal communicator in every pocket is "a pipe dream driven by greed" (link). In hindsight, Sculley was completely right and Grove was completely wrong. ↩
Some references to Intel's "Mother of all demos" are Computer Industry Gathers Amid Chaos, New York Times, Oct 21, 1991 and "Intel's High-Tech Vision of the Future: Chipmaker proposes using computers to dramatically improve productivity", San Francisco Chronicle, Oct 21, 1991, p24. The title of an article in Microprocessor Report, "Intel Declares Victory in the Mother of All Demos" (Nov. 20, 1991), alluded to the recently-ended war. Fortune wrote about Intel's demo in the Feb 17, 1997 issue. A longer description of Intel's demo is in the book Strategy is Destiny. ↩
Several sources claim that Andy van Dam was the first to call Engelbart's demo "The Mother of All Demos." Although van Dam attended the 1968 demo, I couldn't find any evidence that he coined the phrase. John Markoff, a technology journalist for The New York Times, wrote a book What the Dormouse Said: How the Sixties Counterculture Shaped the Personal Computer Industry. In this book, Markoff wrote about Engelbart's demo, saying "Years later, his talk remained 'the mother of all demos' in the words of Andries van Dam, a Brown University computer scientist." As far as I can tell, van Dam used the phrase but only after it had already been popularized by Levy. ↩
It's curious to write that the demonstration was still called the "mother of all demos" when the phrase was just a few years old. ↩
The photo below shows a keyset from the Xerox Alto. The five keys are labeled with separate functions—Copy, Undelete, Move, Draw, and Fine— for use with ALE, a program for IC design. ALE supported keyset chording in combination with the mouse.
After I implemented this interface, I came across a project that constructed a 3D-printed chording keyset, also using a Teensy for the USB interface. You can find that project here. ↩
In 1993, Intel released the high-performance Pentium processor, the start of the long-running Pentium line. I've been examining the Pentium's circuitry in detail and I came across a circuit to multiply by three, a complex circuit with thousands of transistors. Why does the Pentium have a circuit to multiply specifically by three? Why is it so complicated? In this article, I examine this multiplier—which I'll call the ×3 circuit—and explain its purpose and how it is implemented.
It turns out that this multiplier is a small part of the Pentium's floating-point multiplier circuit. In particular, the Pentium multiplies two 64-bit numbers using base-8 multiplication, which is faster than binary multiplication.1 However, multiplying by 3 needs to be handled as a special case. Moreover, since the rest of the multiplication process can't start until the multiplication by 3 finishes, this circuit must be very fast. If you've studied digital design, you may have heard of techniques such as carry lookahead, Kogge-Stone addition, and carry-select addition. I'll explain how the ×3 circuit combines all these techniques to maximize performance.
The photo below shows the Pentium's thumbnail-sized silicon die under a microscope. I've labeled the main functional blocks. In the center is the integer execution unit that performs most instructions. On the left, the code and data caches improve memory performance. The floating point unit, in the lower right, performs floating point operations. Almost half of the floating point unit is occupied by the multiplier, which uses an array of adders to rapidly multiply two 64-bit numbers. The focus of this article is the ×3 circuit, highlighted in yellow near the top of the multiplier. As you can see, the ×3 circuit takes up a nontrivial amount of the Pentium die, especially considering that its task seems simple.
Multiplying two numbers in binary is conceptually straightforward. You can think of binary multiplication as similar to grade-school long multiplication, but with binary numbers instead of decimal numbers. The example below shows how 5×6 is computed in binary: the three terms are added to produce the result. Conveniently, each term is either the multiplicand (101 in this case) or 0, shifted appropriately, so computing the terms is easy.
101
×110
―――
000 i.e. 0×101
101 i.e. 1×101
+101 i.e. 1×101
―――――
11110
Unfortunately, this straightforward multiplication approach is slow. With the three-bit numbers above, there are three terms to add. But if you multiply two 64-bit numbers, you have 64 terms to add, requiring a lot of time and/or circuitry.
The Pentium uses a more complicated approach, computing multiplication in base 8. The idea is to consider the multiplier in groups of three bits, so instead of multiplying by 0 or 1 in each step, you multiply by a number from 0 to 7. Each term that gets added is still in binary, but the number of terms is reduced by a factor of three. Thus, instead of adding 64 terms, you add 22 terms, providing a substantial reduction in the circuitry required. (I'll describe the full details of the Pentium multiplier in a future article.2)
The downside to radix-8 multiplication is that multiplying by a number from 0 to 7 is much more complicated than multiplying by 0 or 1, which is almost trivial. Fortunately, there are some shortcuts. Note that multiplying by 2 is the same as shifting the number to the left by 1 bit position, which is very easy in hardware—you wire each bit one position to the left. Similarly, to multiply by 4, shift the multiplicand two bit positions to the left.
Multiplying by 7 seems inconvenient, but there is a trick, known as Booth's multiplication algorithm. Instead of multiplying by 7, you add 8 times the number and subtract the number, ending up with 7 times the number. You might think this requires two steps, but the trick is to multiply by one more in the (base-8) digit to the left, so you get the factor of 8 without an additional step. (A base-10 analogy is that if you want to multiply by 19, you can multiply by 20 and subtract the multiplicand.) Thus, you can get the ×7 by subtracting. Similarly, for a ×6 term, you can subtract a ×2 multiple and add ×8 in the next digit. Thus, the only difficult multiple is ×3. (What about ×5? If you can compute ×3, you can subtract that from ×8 to get ×5.)
To summarize, the Pentium's radix-8 Booth's algorithm is a fast way to multiply, but it requires a special circuit to produce the ×3 multiple of the multiplicand.
Multiplying a number by three is straightforward in binary: add the number to itself, shifted to the left one position. (As mentioned above, shifting to the left is the same as multiplying by two and is easy in hardware.) Unfortunately, using a simple adder is too slow.
The problem with addition is that carries make addition slow. Consider calculating 99999+1 by hand. You'll start with 9+1=10, then carry the one, generating another carry, which generates another carry, and so forth, until you go through all the digits. Computer addition has the same problem: If you're adding two numbers, the low-order bits can generate a carry that then propagates through all the bits. An adder that works this way—known as a ripple carry adder—will be slow because the carry has to ripple through all the bits. As a result, CPUs use special circuits to make addition faster.
One solution is the carry-lookahead adder. In this adder, all the carry bits are computed in parallel, before computing the sums. Then, the sum bits can be computed in parallel, using the carry bits. As a result, the addition can be completed quickly, without waiting for the carries to ripple through the entire sum.
It may seem impossible to compute the carries without computing the sum first, but there's a way to do it.
For each bit position, you determine signals called "carry generate" and "carry propagate".
These signals can then be used to determine all the carries in parallel.
The generate signal indicates that the position generates a carry. For instance, if you add binary
1xx and 1xx (where x is an arbitrary bit), a carry will be generated from the top bit,
regardless of the unspecified bits.
On the other hand, adding 0xx and 0xx will never generate a carry.
Thus, the generate signal is produced for the first case but not the second.
But what about 1xx plus 0xx? We might get a carry, for instance, 111+001, but we might not,
for instance, 101+001. In this "maybe" case, we set the carry propagate signal, indicating that a carry into the
position will get propagated out of the position. For example, if there is a carry out of
the middle position, 1xx+0xx will have a carry from the top bit. But if there is no carry out of the middle position, then
there will not be a carry from the top bit. In other words, the propagate signal indicates that a carry into the top bit will be propagated out of the top
bit.
To summarize, adding 1+1 will generate a carry. Adding 0+1 or 1+0 will propagate a
carry.
Thus, the generate signal is formed at each position by Gn = An·Bn, where A and B are the inputs.
The propagate signal is Pn = An+Bn,
the logical-OR of the inputs.3
Now that the propagate and generate signals are defined, some moderately complex logic4 can compute the carry Cn into each bit position. The important thing is that all the carry bits can be computed in parallel, without waiting for the carry to ripple through each bit position. Once each carry is computed, the sum bits can be computed in parallel: Sn = An ⊕ Bn ⊕ Cn. In other words, the two input bits and the computed carry are combined with exclusive-or. Thus, the entire sum can be computed in parallel by using carry lookahead. However, there are complications.
The carry bits can be generated directly from the G and P signals. However, the straightforward approach requires too much hardware as the number of bits increases. Moreover, this approach needs gates with many inputs, which are slow for electrical reasons. For these reasons, the Pentium uses two techniques to keep the hardware requirements for carry lookahead tractable. First, it uses a "parallel prefix adder" algorithm for carry lookahead across 8-bit chunks.7 Second, it uses a two-level hierarchical approach for carry lookahead: the upper carry-lookahead circuit handles eight 8-bit chunks, using the same 8-bit algorithm.5
The photo below shows the complete ×3 circuit; you can see that the circuitry is divided into blocks of 8 bits. (Although I'm calling this a 64-bit circuit, it really produces a 69-bit output: there are 5 "extra" bits on the left to avoid overflow and to provide additional bits for rounding.)
The idea of the parallel-prefix adder is to
produce the propagate and generate signals across ranges of bits, not just single bits as before.
For instance, the propagate signal P32 indicates that a carry in to bit 2 would be propagated out of bit 3,
(This would happen with 10xx+01xx, for example.)
And G30 indicates that bits 3 to 0 generate a carry out of bit 3.
(This would happen with 1011+0111, for example.)
Using some mathematical tricks,6 you can take the P and G values for two smaller ranges and merge them into the P and G values for the combined range. For instance, you can start with the P and G values for bits 0 and 1, and produce P10 and G10, the propagate and generate signals describing two bits. These could be merged with P32 and G32 to produce P30 and G30, indicating if a carry is propagated across bits 3-0 or generated by bits 3-0. Note that Gn0 tells us if a carry is generated into bit n+1 from all the lower bits, which is the Cn+1 carry value that we need to compute the final sum. This merging process is more efficient than the "brute force" implementation of the carry-lookahead logic since logic subexpressions can be reused.
There are many different ways that you can combine the P and G terms to generate the necessary terms.8 The Pentium uses an approach called Kogge-Stone that attempts to minimize the total delay while keeping the amount of circuitry reasonable. The diagram below is the standard diagram that illustrates how a Kogge-Stone adder works. It's rather abstract, but I'll try to explain it. The diagram shows how the P and G signals are merged to produce each output at the bottom. Each square box at the top generates the P and G signals for that bit. Each line corresponds to both the P and the G signal. Each diamond combines two ranges of P and G signals to generate new P and G signals for the combined range. Thus, the signals cover wider ranges of bits as they progress downward, ending with the Gn0 outputs that indicate carries.
I've labeled a few of the intermediate signals so you can get an idea of how it works. Circuit "A" combines P7 and G7 with P6 and G6 to produce the signals describing two bits: P76 and G76. Similarly, circuit "B" combines P76 and G76 with P54 and G54 to produce the signals describing four bits: P74 and G74. Finally, circuit "C" produces the final outputs for bit 7: P70 and G70. Note that most of the intermediate results are used twice, reducing the amount of circuitry. Moreover, there are at most three levels of combination circuitry, reducing the delay compared to a deeper network.
The key point is the P and G values are computed in parallel so the carry bits can all be computed in parallel, without waiting for the carry to ripple through all the bits. (If this explanation doesn't make sense, see my discussion of the Kogge-Stone adder in the Pentium's division circuit for a different—but maybe still confusing—explanation.)
The Kogge-Stone approach can be extended to 64 bits, but the amount of circuitry and wiring becomes overwhelming. Instead, the Pentium uses a recursive, hierarchical approach with two levels of Kogge-Stone lookahead. The lower layer uses eight Kogge-Stone adders as described above, supporting 64 bits in total.
The upper layer uses a single eight-bit Kogge-Stone lookahead circuit, treating each of the lower chunks as a single bit. That is, a lower chunk has a propagate signal P indicating that a carry into the chunk will be propagated out, as well as a generate signal G indicating that the chunk generates a carry. The upper Kogge-Stone circuit combines these chunked signals to determine if carries will be generated or propagated by groups of chunks.9
To summarize, each of the eight lower lookahead circuits computes the carries within an 8-bit chunk. The upper lookahead circuit computes the carries into and out of each 8-bit chunk. In combination, the circuits rapidly provide all the carries needed to compute the 64-bit sum.
Suppose you're on a game show: "What is 553 + 246 + c? In 10 seconds, I'll tell you if c is 0 or 1 and whoever gives the answer first wins $1000." Obviously, you shouldn't just sit around until you get c. You should do the two sums now, so you can hit the buzzer as soon as c is announced. This is the concept behind the carry-select adder: perform two additions—with a carry-in and without--and then supply the correct answer as soon as the carry is available. The carry-select adder requires additional hardware—two adders along with a multiplexer to select the result—but it overlaps the time to compute the sum with the time to compute the carry. In effect, the addition and the carry lookahead operations are performed in parallel, with the multiplexer combining the results from each.
The Pentium uses a carry-select adder for each 8-bit chunk in the ×3 circuit. The carry from the second-level carry-lookahead selects which sum should be produced for the chunk. Thus, the time to compute the carry is overlapped with the time to compute the sum.
The image below zooms in on an 8-bit chunk of the ×3 multiplier, implementing an 8-bit adder. Eight input lines are at the top (along with some unrelated wires). Note that each input line splits with a signal going to the adder on the left and a signal going to the right. This is what causes the adder to multiply by 3: it adds the input and the input shifted one bit to the left, i.e. multiplied by two. The top part of the adder has eight circuits to produce the propagate and generate signals. These signals go into the 8-bit Kogge-Stone lookahead circuit. Although most of the adder consists of a circuit block repeated eight times, the Kogge-Stone circuitry appears chaotic. This is because each bit of the Kogge-Stone circuit is different—higher bits are more complicated to compute than lower bits.
The lower half of the circuit block contains an 8-bit carry-select adder. This circuit produces two sums, with multiplexers selecting the correct sum based on the carry into the block. Note that the carry-select adder blocks are narrower than the other circuitry.10 This makes room for a Kogge-Stone block on the left. The second level Kogge-Stone circuitry is split up; the 8-bit carry-lookahead circuitry has one bit implemented in each block of the adder, and produces the carry-in signal for that adder block. In other words, the image above includes 1/8 of the second-level Kogge-Stone circuit. Finally, eight driver circuits amplify the output bits before they are sent to the rest of the floating-point multiplier.
The block diagram below shows the pieces are combined to form the ×3 multiplier. The multiplier has eight 8-bit adder blocks (green boxes, corresponding to the image above). Each block computes eight bits of the total sum. Each block provides P70 and G70 signals to the second-level lookahead, which determines if each block receives a carry in. The key point to this architecture is that everything is computed in parallel, making the addition fast.
In the diagram above, the first 8-bit block is expanded to show its contents. The 8-bit lookahead circuit generates the P and G signals that determine the internal carry signals. The carry-select adder contains two 8-bit adders that use the carry lookahead values. As described earlier, one adder assumes that the block's carry-in is 1 and the second assumes the carry-in is 0. When the real carry in value is provided by the second-level lookahead circuit, the multiplexer selects the correct sum.
The photo below shows how the complete multiplier is constructed from 8-bit blocks. The multiplier produces a 69-bit output; there are 5 "extra" bits on the left. Note that the second-level Kogge-Stone blocks are larger on the right than the left since the lookahead circuitry is more complex for higher-order bits.
Going back to the full ×3 circuit above, you can see that the 8 bits on the right have significantly simpler circuitry. Because there is no carry-in to this block, the carry-select circuitry can be omitted. The block's internal carries, generated by the Kogge-Stone lookahead circuitry, are added using exclusive-NOR gates. The diagram below shows the implementation of an XNOR gate, using inverters and a multiplexer.
I'll now describe one of the multiplier's circuits at the transistor level, in particular an XNOR gate. It's interesting to look at XNOR because XNOR (like XOR) is a tricky gate to implement and different processors use very different approaches. For instance, the Intel 386 implements XOR from AND-NOR gates (details) while the Z-80 uses pass transistors (details). The Pentium, on the other hand, uses a multiplexer.
The diagram above shows one of the XNOR gates in the adder's low bits.11 The gate is constructed from four inverters and a pass-transistor multiplexer. Input B selects one of the multiplexer's two inputs: input A or input A inverted. The result is the XNOR function. (Inverter 1 buffers the input, inverter 5 buffers the output, and inverter 4 provides the complemented B signal to drive the multiplexer.)
For the photo, I removed the top two metal layers from the chip, leaving the bottom metal layer, called M1. The doped silicon regions are barely visible beneath the metal. When a polysilicon line crosses doped silicon, it forms the gate of a transistor. This CMOS circuit has NMOS transistors at the top and PMOS transistors at the bottom. Each inverter consists of two transistors, while the multiplexer consists of four transistors.
The outputs from the ×3 circuit require high current. In particular, each signal from the ×3 circuit can drive up to 22 terms in the floating-point multiplier. Moreover, the destination circuits can be a significant distance from the ×3 circuit due to the size of the multiplier. Since the ×3 signals are connected to many transistor gates through long wires, the capacitance is high, requiring high current to change the signals quickly.
The Pentium is constructed with a somewhat unusual process called BiCMOS, which combines bipolar transistors and CMOS on the same chip. The Pentium extensively uses BiCMOS circuits since they reduced signal delays by up to 35%. Intel also used BiCMOS for the Pentium Pro, Pentium II, Pentium III, and Xeon processors. However, as chip voltages dropped, the benefit from bipolar transistors dropped too and BiCMOS was eventually abandoned.
The schematic below shows a simplified BiCMOS driver that inverts its input. A 0 input turns on the upper inverter, providing current into the bipolar (NPN) transistor's base. This turns on the transistor, causing it to pull the output high strongly and rapidly. A 1 input, on the other hand, will stop the current flow through the NPN transistor's base, turning it off. At the same time, the lower inverter will pull the output low. (The NPN transistor can only pull the output high.)
Note the asymmetrical construction of the inverters. Since the upper inverter must provide a large current into the NPN transistor's base, it is designed to produce a strong (high-current) positive output and a weak low output. The lower inverter, on the other hand, is responsible for pulling the output low. Thus, it is constructed to produce a strong low output, while the high output can be weak.
The driver of the ×3 circuit goes one step further: it uses a BiCMOS driver to drive a second BiCMOS driver. The motivation is that the high-current inverters have fairly large transistor gates, so they need to be driven with high current (but not as much as they produce, so there isn't an infinite regress).12
The schematic below shows the BiCMOS driver circuit that the ×3 multiplier uses. Note the large, box-like appearance of the NPN transistors, very different from the regular MOS transistors. Each box contains two NPN transistors sharing collectors: a larger transistor on the left and a smaller one on the right. You might expect these transistors to work together, but the contiguous transistors are part of two separate circuits. Instead, the small NPN transistor to the left and the large NPN transistor to the right are part of the same circuit.
The inverters are constructed as standard CMOS circuits with PMOS transistors to pull the output high and NMOS transistors to pull the output low. The inverters are carefully structured to provide asymmetrical current, making them more interesting than typical inverters. Two pullup transistors have a long gate, making these transistors unusually weak. Other parts of the inverters have multiple transistors in parallel, providing more current. Moreover, the inverters have unusual layouts, with the NMOS and PMOS transistors widely separated to make the layout more efficient. For more on BiCMOS in the Pentium, see my article on interesting BiCMOS circuits in the Pentium.
Hardware support for computer multiplication has a long history going back to the 1950s.13 Early microprocessors, though, had very limited capabilities, so microprocessors such as the 6502 didn't have hardware support for multiplication; users had to implement multiplication in software through shifts and adds. As hardware advanced, processors provided multiplication instructions but they were still slow. For example, the Intel 8086 processor (1978) implemented multiplication in microcode, performing a slow shift-and-add loop internally. Processors became exponentially more powerful over time, as described by Moore's Law, allowing later processors to include dedicated multiplication hardware. The 386 processor (1985) included a multiply unit, but it was still slow, taking up to 41 clock cycles for a multiplication instruction.
By the time of the Pentium (1993), microprocessors contained millions of transistors, opening up new possibilities for design. With a seemingly unlimited number of transistors, chip architects could look at complicated new approaches to squeeze more performance out of a system. This ×3 multiplier contains roughly 9000 transistors, a bit more than an entire Z80 microprocessor (1976). Keep in mind that the ×3 multiplier is a small part of the floating-point multiplier, which is part of the floating-point unit in the Pentium. Thus, this small piece of a feature is more complicated than an entire microprocessor from 17 years earlier, illustrating the incredible growth in processor complexity.
I plan to write more about the implementation of the Pentium, so follow me on Bluesky (@righto.com) or RSS for updates. (I'm no longer on Twitter.) The Pentium Navajo rug inspired me to examine the Pentium in more detail.
A floating-point multiplication on the Pentium takes three clock cycles, of which the multiplication circuitry is busy for two cycles.
(See Agner Fog's optimization manual.)
In comparison, integer multiplication (MUL) is much slower, taking 11 cycles.
The Nehalem microarchitecture (2008) reduced floating-point multiplication time to 1 cycle. ↩
I'll give a quick outline of the Pentium's floating-point multiplier as a preview. The multiplier is built from a tree of ten carry-save adders to sum the terms. Each carry-save adder is a 4:2 compression adder, taking four input bits and producing two output bits. The output from the carry-save adder is converted to the final result by an adder using Kogge-Stone lookahead and carry select. Multiplying two 64-bit numbers yields 128 bits, but the Pentium produces a 64-bit result. (There are actually a few more bits for rounding.) The low 64 bits can't simply be discarded because they could produce a carry into the preserved bits. Thus, the low 64 bits go into another Kogge-Stone lookahead circuit that doesn't produce a sum, but indicates if there is a carry. Since the datapath is 64 bits wide, but the product is 128 bits, there are many shift stages to move the bits to the right column. Moreover, the adders are somewhat wider than 64 bits as needed to hold the intermediate sums. ↩
The bits 1+1 will set generate, but should propagate be set too?
It doesn't make a difference as far as the equations. This adder sets propagate for 1+1 but some
other adders do not.
The answer depends on if you use an inclusive-or or exclusive-or gate
to produce the propagate signal. ↩
The carry Cn at each bit position n can be computed from the G and P signals by considering the various cases:
C1 = G0: a carry into bit 1 occurs if a carry is generated from bit 0.
C2 = G1 + G0P1: A carry into bit 2 occurs if bit 1 generates a carry or bit 1 propagates a carry from bit 0.
C3 = G2 + G1P2 + G0P1P2: A carry into bit 3 occurs if bit 2 generates a carry, or bit 2 propagates a carry generated from bit 1, or bits 2 and 1 propagate a carry generated from bit 0.
C4 = G3 + G2P3 + G1P2P3 + G0P1P2P3: A carry into bit 4 occurs if a carry is generated from bit 3, 2, 1, or 0 along with the necessary propagate signals.
And so on...
Note that the formula gets more complicated for each bit position. The circuit complexity is approximately O(N3), depending on how you measure it. Thus, implementing the carry lookahead formula directly becomes impractical as the number of bits gets large. The Kogge-Stone approach uses approximately O(N log N) transistors, but the wiring becomes excessive for large N since there are N/2 wires of length N/2. Using a tree of Kogge-Stone circuits reduces the amount of wiring. ↩
The 8-bit chunks in the circuitry have nothing to do with bytes. The motivation is that 8 bits is a reasonable size for a chunk, as well as providing a nice breakdown into 8 chunks of 8 bits. Other systems have used 4-bit chunks for carry lookahead (such as minicomputers based on the 74181 ALU chip). ↩
I won't go into the mathematics of merging P and G signals; see, for example, Adder Circuits or Carry Lookahead Adders for additional details. The important factor is that the carry merge operator is associative (actually a monoid), so the sub-ranges can be merged in any order. This flexibility is what allows different algorithms with different tradeoffs. ↩
The idea behind a prefix adder is that we want to see if there is a carry out of bit 0, bits 0-1, bits 0-2, bits 0-3, 0-4, and so forth. These are all the prefixes of the word. Since the prefixes are computed in parallel, it's called a parallel prefix adder. ↩
The lookahead merging process can be implemented in many ways, including Kogge-Stone, Brent-Kung, and Ladner-Fischer, with different tradeoffs. For one example, the diagram below shows that Brent-Kung uses fewer "diamonds" but more layers. Thus, a Brent-Kung adder uses less circuitry but is slower. (You can follow each output upward to verify that the tree reaches the correct inputs.)
The higher-level Kogge-Stone lookahead circuit uses the eight P70 and G70 signals from the eight lower-level lookahead circuits. Note that P70 and G70 indicate that an 8-bit chunk will propagate or generate a carry. The higher-level lookahead circuit treats 8-bit chunks as a unit, while the lower-level lookahead circuit treats 1-bit chunks as a unit. Thus, the higher-level and lower-level lookahead circuits are essentially identical, acting on 8-bit values. ↩
The floating-point unit is built from fixed-width columns, one for each bit. Each column is 38.5 µm wide, so the circuitry in each column must be designed to fit that width. For the most part, the same circuitry is repeated for each of the 64 (or so) bits. The carry-select adder is unusual since it doesn't follow the column width of the rest of the floating-point unit. Instead, it crams 8 circuits into the width of 6.5 regular circuits. This leaves room for one Kogge-Stone circuitry block. ↩
Because there is no carry-in to the lowest 8-bit block of the ×3 circuit, the carry-select circuit is not needed. Instead, each output bit can be computed using an XNOR gate. ↩
The principle of Logical Effort explains that for best performance, you don't want to jump from a small signal to a high-current signal in one step. Instead, a small signal produces a medium signal, which produces a larger signal. By using multiple stages of circuitry, the overall delay can be reduced. ↩
The Booth multiplication technique was described in 1951, while parallel multipliers were proposed in the mid-1960s by Wallace and Dadda. Jumping ahead to higher-radix multiplication, a 1992 paper A Fast Hybrid Multiplier Combining Booth and Wallace/Dadda Algorithms from Motorola discusses radix-4 and radix-8 algorithms for a 32-bit multiplier, but decides that computing the ×3 multiple makes radix-8 impractical. IBM discussed a 32-bit multiplier in 1997: A Radix-8 CMOS S/390 Multiplier. Bewick's 1994 PhD thesis Fast Multiplication: Algorithms and Implementation describes numerous algorithms.
For adders, Two-Operand Addition is an interesting presentation on different approaches. CMOS VLSI Design has a good discussion of addition and various lookahead networks. It summarizes the tradeoffs: "Brent-Kung has too many logic levels. Sklansky has too much fanout. And Kogge-Stone has too many wires. Between these three extremes, the Han-Carlson, Ladner-Fischer, and Knowles trees fill out the design space with different compromises between number of stages, fanout, and wire count." The approach used in the Pentium's ×3 multiplier is sometimes called a sparse-tree adder. ↩
What is the origin of the word "mainframe", referring to a large, complex computer? Most sources agree that the term is related to the frames that held early computers, but the details are vague.1 It turns out that the history is more interesting and complicated than you'd expect.
Based on my research, the earliest computer to use the term "main frame" was the IBM 701 computer (1952), which consisted of boxes called "frames." The 701 system consisted of two power frames, a power distribution frame, an electrostatic storage frame, a drum frame, tape frames, and most importantly a main frame. The IBM 701's main frame is shown in the documentation below.2
The meaning of "mainframe" has evolved, shifting from being a part of a computer to being a type of computer. For decades, "mainframe" referred to the physical box of the computer; unlike modern usage, this "mainframe" could be a minicomputer or even microcomputer. Simultaneously, "mainframe" was a synonym for "central processing unit." In the 1970s, the modern meaning started to develop—a large, powerful computer for transaction processing or business applications—but it took decades for this meaning to replace the earlier ones. In this article, I'll examine the history of these shifting meanings in detail.
Early computers used a variety of mounting and packaging techniques including panels, cabinets, racks, and bays.3 This packaging made it very difficult to install or move a computer, often requiring cranes or the removal of walls.4 To avoid these problems, the designers of the IBM 701 computer came up with an innovative packaging technique. This computer was constructed as individual units that would pass through a standard doorway, would fit on a standard elevator, and could be transported with normal trucking or aircraft facilities.7 These units were built from a metal frame with covers attached, so each unit was called a frame. The frames were named according to their function, such as the power frames and the tape frame. Naturally, the main part of the computer was called the main frame.
The IBM 701's internal documentation used "main frame" frequently to indicate the main box of the computer, alongside "power frame", "core frame", and so forth. For instance, each component in the schematics was labeled with its location in the computer, "MF" for the main frame.6 Externally, however, IBM documentation described the parts of the 701 computer as units rather than frames.5
The term "main frame" was used by a few other computers in the 1950s.8 For instance, the JOHNNIAC Progress Report (August 8, 1952) mentions that "the main frame for the JOHNNIAC is ready to receive registers" and they could test the arithmetic unit "in the JOHNNIAC main frame in October."10 An article on the RAND Computer in 1953 stated that "The main frame is completed and partially wired" The main body of a computer called ERMA is labeled "main frame" in the 1955 Proceedings of the Eastern Computer Conference.9
The progression of the word "main frame" can be seen in reports from the Ballistics Research Lab (BRL) that list almost all the computers in the United States. In the 1955 BRL report, most computers were built from cabinets or racks; the phrase "main frame" was only used with the IBM 650, 701, and 704. By 1961, the BRL report shows "main frame" appearing in descriptions of the IBM 702, 705, 709, and 650 RAMAC, as well as the Univac FILE 0, FILE I, RCA 501, READIX, and Teleregister Telefile. This shows that the use of "main frame" was increasing, but still mostly an IBM term.
In modern usage, mainframes are distinct from minicomputers or microcomputers. But until the 1980s, the word "mainframe" could also mean the main physical part of a minicomputer or microcomputer. For instance, a "minicomputer mainframe" was not a powerful minicomputer, but simply the main part of a minicomputer.13 For example, the PDP-11 is an iconic minicomputer, but DEC discussed its "mainframe."14. Similarly, the desktop-sized HP 2115A and Varian Data 620i computers also had mainframes.15 As late as 1981, the book Mini and Microcomputers mentioned "a minicomputer mainframe."
Even microcomputers had a mainframe: the cover of Radio Electronics (1978, above) stated, "Own your own Personal Computer: Mainframes for Hobbyists", using the definition below. An article "Introduction to Personal Computers" in Radio Electronics (Mar 1979) uses a similar meaning: "The first choice you will have to make is the mainframe or actual enclosure that the computer will sit in." The popular hobbyist magazine BYTE also used "mainframe" to describe a microprocessor's box in the 1970s and early 1980s16. BYTE sometimes used the word "mainframe" both to describe a large IBM computer and to describe a home computer box in the same issue, illustrating that the two distinct meanings coexisted.
Words often change meaning through metonymy, where a word takes on the meaning of something closely associated with the original meaning. Through this process, "main frame" shifted from the physical frame (as a box) to the functional contents of the frame, specifically the central processing unit.17
The earliest instance that I could find of the "main frame" being equated with the central processing unit was in 1955. Survey of Data Processors stated: "The central processing unit is known by other names; the arithmetic and ligical [sic] unit, the main frame, the computer, etc. but we shall refer to it, usually, as the central processing unit." A similar definition appeared in Radio Electronics (June 1957, p37): "These arithmetic operations are performed in what is called the arithmetic unit of the machine, also sometimes referred to as the 'main frame.'"
The US Department of Agriculture's Glossary of ADP Terminology (1960) uses the definition: "MAIN FRAME - The central processor of the computer system. It contains the main memory, arithmetic unit and special register groups." I'll mention that "special register groups" is nonsense that was repeated for years.18 This definition was reused and extended in the government's Automatic Data Processing Glossary, published in 1962 "for use as an authoritative reference by all officials and employees of the executive branch of the Government" (below). This definition was reused in many other places, notably the Oxford English Dictionary.19
By the early 1980s, defining a mainframe as the CPU had become obsolete. IBM stated that "mainframe" was a deprecated term for "processing unit" in the Vocabulary for Data Processing, Telecommunications, and Office Systems (1981); the American National Dictionary for Information Processing Systems (1982) was similar. Computers and Business Information Processing (1983) bluntly stated: "According to the official definition, 'mainframe' and 'CPU' are synonyms. Nobody uses the word mainframe that way."
Rather than defining the mainframe as the CPU, some dictionaries defined the mainframe in opposition to the "peripherals", the computer's I/O devices. The two definitions are essentially the same, but have a different focus.20 One example is the IFIP-ICC Vocabulary of Information Processing (1966) which defined "central processor" and "main frame" as "that part of an automatic data processing system which is not considered as peripheral equipment." Computer Dictionary (1982) had the definition "main frame—The fundamental portion of a computer, i.e. the portion that contains the CPU and control elements of a computer system, as contrasted with peripheral or remote devices usually of an input-output or memory nature."
One reason for this definition was that computer usage was billed for mainframe time, while other tasks such as printing results could save money by taking place directly on the peripherals without using the mainframe itself.21 A second reason was that the mainframe vs. peripheral split mirrored the composition of the computer industry, especially in the late 1960s and 1970s. Computer systems were built by a handful of companies, led by IBM. Compatible I/O devices and memory were built by many other companies that could sell them at a lower cost than IBM.22 Publications about the computer industry needed convenient terms to describe these two industry sectors, and they often used "mainframe manufacturers" and "peripheral manufacturers."
An interesting linguistic shift is from "main frame" as two independent words to a compound word: either hyphenated "main-frame" or the single word "mainframe." This indicates the change from "main frame" being a type of frame to "mainframe" being a new concept. The earliest instance of hyphenated "main-frame" that I found was from 1959 in IBM Information Retrieval Systems Conference. "Mainframe" as a single, non-hyphenated word appears the same year in Datamation, mentioning the mainframe of the NEAC2201 computer. In 1962, the IBM 7090 Installation Instructions refer to a "Mainframe Diag[nostic] and Reliability Program." (Curiously, the document also uses "main frame" as two words in several places.) The 1962 book Information Retrieval Management discusses how much computer time document queries can take: "A run of 100 or more machine questions may require two to five minutes of mainframe time." This shows that by 1962, "main frame" had semantically shifted to a new word, "mainframe."
So far, I've shown how "mainframe" started as a physical frame in the computer, and then was generalized to describe the CPU. But how did "mainframe" change from being part of a computer to being a class of computers? This was a gradual process, largely happening in the mid-1970s as the rise of the minicomputer and microcomputer created a need for a word to describe large computers.
Although microcomputers, minicomputers, and mainframes are now viewed as distinct categories, this was not the case at first. For instance, a 1966 computer buyer's guide lumps together computers ranging from desk-sized to 70,000 square feet.23 Around 1968, however, the term "minicomputer" was created to describe small computers. The story is that the head of DEC in England created the term, inspired by the miniskirt and the Mini Minor car.24 While minicomputers had a specific name, larger computers did not.25
Gradually in the 1970s "mainframe" came to be a separate category, distinct from "minicomputer."2627 An early example is Datamation (1970), describing systems of various sizes: "mainframe, minicomputer, data logger, converters, readers and sorters, terminals." The influential business report EDP first split mainframes from minicomputers in 1972.28 The line between minicomputers and mainframes was controversial, with articles such as Distinction Helpful for Minis, Mainframes and Micro, Mini, or Mainframe? Confusion persists (1981) attempting to clarify the issue.29
With the development of the microprocessor, computers became categorized as mainframes, minicomputers or microcomputers. For instance, a 1975 Computerworld article discussed how the minicomputer competes against the microcomputer and mainframes. Adam Osborne's An Introduction to Microcomputers (1977) described computers as divided into mainframes, minicomputers, and microcomputers by price, power, and size. He pointed out the large overlap between categories and avoided specific definitions, stating that "A minicomputer is a minicomputer, and a mainframe is a mainframe, because that is what the manufacturer calls it."32
In the late 1980s, computer industry dictionaries started defining a mainframe as a large computer, often explicitly contrasted with a minicomputer or microcomputer. By 1990, they mentioned the networked aspects of mainframes.33
Even though IBM is almost synonymous with "mainframe" now, IBM avoided marketing use of the word for many years, preferring terms such as "general-purpose computer."35 IBM's book Planning a Computer System (1962) repeatedly referred to "general-purpose computers" and "large-scale computers", but never used the word "mainframe."34 The announcement of the revolutionary System/360 (1964) didn't use the word "mainframe"; it was called a general-purpose computer system. The announcement of the System/370 (1970) discussed "medium- and large-scale systems." The System/32 introduction (1977) said, "System/32 is a general purpose computer..." The 1982 announcement of the 3084, IBM's most powerful computer at the time, called it a "large scale processor" not a mainframe.
IBM started using "mainframe" as a marketing term in the mid-1980s. For example, the 3270 PC Guide (1986) refers to "IBM mainframe computers." An IBM 9370 Information System brochure (c. 1986) says the system was "designed to provide mainframe power." IBM's brochure for the 3090 processor (1987) called them "advanced general-purpose computers" but also mentioned "mainframe computers." A System 390 brochure (c. 1990) discussed "entry into the mainframe class." The 1990 announcement of the ES/9000 called them "the most powerful mainframe systems the company has ever offered."
By 2000, IBM had enthusiastically adopted the mainframe label: the z900 announcement used the word "mainframe" six times, calling it the "reinvented mainframe." In 2003, IBM announced "The Mainframe Charter", describing IBM's "mainframe values" and "mainframe strategy." Now, IBM has retroactively applied the name "mainframe" to their large computers going back to 1959 (link), (link).
While "mainframe" was a relatively obscure computer term for many years, it became widespread in the 1980s. The Google Ngram graph below shows the popularity of "microcomputer", "minicomputer", and "mainframe" in books.36 The terms became popular during the late 1970s and 1980s. The popularity of "minicomputer" and "microcomputer" roughly mirrored the development of these classes of computers. Unexpectedly, even though mainframes were the earliest computers, the term "mainframe" peaked later than the other types of computers.
I studied many old dictionaries to see when the word "mainframe" showed up and how they defined it. To summarize, "mainframe" started to appear in dictionaries in the late 1970s, first defining the mainframe in opposition to peripherals or as the CPU. In the 1980s, the definition gradually changed to the modern definition, with a mainframe distinguished as being large, fast, and often centralized system. These definitions were roughly a decade behind industry usage, which switched to the modern meaning in the 1970s.
The word didn't appear in older dictionaries, such as the Random House College Dictionary (1968) and Merriam-Webster (1974). The earliest definition I could find was in the supplement to Webster's International Dictionary (1976): "a computer and esp. the computer itself and its cabinet as distinguished from peripheral devices connected with it." Similar definitions appeared in Webster's New Collegiate Dictionary (1976, 1980).
A CPU-based definition appeared in Random House College Dictionary (1980): "the device within a computer which contains the central control and arithmetic units, responsible for the essential control and computational functions. Also called central processing unit." The Random House Dictionary (1978, 1988 printing) was similar. The American Heritage Dictionary (1982, 1985) combined the CPU and peripheral approaches: "mainframe. The central processing unit of a computer exclusive of peripheral and remote devices."
The modern definition as a large computer appeared alongside the old definition in Webster's Ninth New Collegiate Dictionary (1983): "mainframe (1964): a computer with its cabinet and internal circuits; also: a large fast computer that can handle multiple tasks concurrently." Only the modern definition appears in The New Merriram-Webster Dictionary (1989): "large fast computer", while Webster's Unabridged Dictionary of the English Language (1989): "mainframe. a large high-speed computer with greater storage capacity than a minicomputer, often serving as the central unit in a system of smaller computers. [MAIN + FRAME]." Random House Webster's College Dictionary (1991) and Random House College Dictionary (2001) had similar definitions.
The Oxford English Dictionary is the principal historical dictionary, so it is interesting to see its view. The 1989 OED gave historical definitions as well as defining mainframe as "any large or general-purpose computer, exp. one supporting numerous peripherals or subordinate computers." It has seven historical examples from 1964 to 1984; the earliest is the 1964 Honeywell Glossary. It quotes a 1970 Dictionary of Computers as saying that the word "Originally implied the main framework of a central processing unit on which the arithmetic unit and associated logic circuits were mounted, but now used colloquially to refer to the central processor itself." The OED also cited a Hewlett-Packard ad from 1974 that used the word "mainframe", but I consider this a mistake as the usage is completely different.15
A look at encyclopedias shows that the word "mainframe" started appearing in discussions of computers in the early 1980s, later than in dictionaries. At the beginning of the 1980s, many encyclopedias focused on large computers, without using the word "mainframe", for instance, The Concise Encyclopedia of the Sciences (1980) and World Book (1980). The word "mainframe" started to appear in supplements such as Britannica Book of the Year (1980) and World Book Year Book (1981), at the same time as they started discussing microcomputers. Soon encyclopedias were using the word "mainframe", for example, Funk & Wagnalls Encyclopedia (1983), Encyclopedia Americana (1983), and World Book (1984). By 1986, even the Doubleday Children's Almanac showed a "mainframe computer."
I examined old newspapers to track the usage of the word "mainframe." The graph below shows the usage of "mainframe" in newspapers. The curve shows a rise in popularity through the 1980s and a steep drop in the late 1990s. The newspaper graph roughly matches the book graph above, although newspapers show a much steeper drop in the late 1990s. Perhaps mainframes aren't in the news anymore, but people still write books about them.
The first newspaper appearances were in classified ads seeking employees, for instance, a 1960 ad in the San Francisco Examiner for people "to monitor and control main-frame operations of electronic computers...and to operate peripheral equipment..." and a (sexist) 1966 ad in the Philadelphia Inquirer for "men with Digital Computer Bkgrnd [sic] (Peripheral or Mainframe)."37
By 1970, "mainframe" started to appear in news articles, for example, "The computer can't work without the mainframe unit." By 1971, the usage increased with phrases such as "mainframe central processor" and "'main-frame' computer manufacturers". 1972 had usages such as "the mainframe or central processing unit is the heart of any computer, and does all the calculations". A 1975 article explained "'Mainframe' is the industry's word for the computer itself, as opposed to associated items such as printers, which are referred to as 'peripherals.'" By 1980, minicomputers and microcomputers were appearing: "All hardware categories-mainframes, minicomputers, microcomputers, and terminals" and "The mainframe and the minis are interconnected."
By 1985, the mainframe was a type of computer, not just the CPU: "These days it's tough to even define 'mainframe'. One definition is that it has for its electronic brain a central processor unit (CPU) that can handle at least 32 bits of information at once. ... A better distinction is that mainframes have numerous processors so they can work on several jobs at once." Articles also discussed "the micro's challenge to the mainframe" and asked, "buy a mainframe, rather than a mini?"
By 1990, descriptions of mainframes became florid: "huge machines laboring away in glass-walled rooms", "the big burner which carries the whole computing load for an organization", "behemoth data crunchers", "the room-size machines that dominated computing until the 1980s", "the giant workhorses that form the nucleus of many data-processing centers", "But it is not raw central-processing-power that makes a mainframe a mainframe. Mainframe computers command their much higher prices because they have much more sophisticated input/output systems."
After extensive searches through archival documents, I found usages of the term "main frame" dating back to 1952, much earlier than previously reported. In particular, the introduction of frames to package the IBM 701 computer led to the use of the word "main frame" for that computer and later ones. The term went through various shades of meaning and remained fairly obscure for many years. In the mid-1970s, the term started describing a large computer, essentially its modern meaning. In the 1980s, the term escaped the computer industry and appeared in dictionaries, encyclopedias, and newspapers. After peaking in the 1990s, the term declined in usage (tracking the decline in mainframe computers), but the term and the mainframe computer both survive.
Two factors drove the popularity of the word "mainframe" in the 1980s with its current meaning of a large computer. First, the terms "microcomputer" and "minicomputer" led to linguistic pressure for a parallel term for large computers. For instance, the business press needed a word to describe IBM and other large computer manufacturers. While "server" is the modern term, "mainframe" easily filled the role back then and was nicely alliterative with "microcomputer" and "minicomputer."38
Second, up until the 1980s, the prototype meaning for "computer" was a large mainframe, typically IBM.39 But as millions of home computers were sold in the early 1980s, the prototypical "computer" shifted to smaller machines. This left a need for a term for large computers, and "mainframe" filled that need. In other words, if you were talking about a large computer in the 1970s, you could say "computer" and people would assume you meant a mainframe. But if you said "computer" in the 1980s, you needed to clarify if it was a large computer.
The word "mainframe" is almost 75 years old and both the computer and the word have gone through extensive changes in this time. The "death of the mainframe" has been proclaimed for well over 30 years but mainframes are still hanging on. Who knows what meaning "mainframe" will have in another 75 years?
Follow me on Bluesky (@righto.com) or RSS. (I'm no longer on Twitter.) Thanks to the Computer History Museum and archivist Sara Lott for access to many documents.
The Computer History Museum states: "Why are they called “Mainframes”? Nobody knows for sure. There was no mainframe “inventor” who coined the term. Probably “main frame” originally referred to the frames (designed for telephone switches) holding processor circuits and main memory, separate from racks or cabinets holding other components. Over time, main frame became mainframe and came to mean 'big computer.'" (Based on my research, I don't think telephone switches have any connection to computer mainframes.)
Several sources explain that the mainframe is named after the frame used to construct the computer. The Jargon File has a long discussion, stating that the term "originally referring to the cabinet containing the central processor unit or ‘main frame’." Ken Uston's Illustrated Guide to the IBM PC (1984) has the definition "MAIN FRAME A large, high-capacity computer, so named because the CPU of this kind of computer used to be mounted on a frame." IBM states that mainframe "Originally referred to the central processing unit of a large computer, which occupied the largest or central frame (rack)." The Microsoft Computer Dictionary (2002) states that the name mainframe "is derived from 'main frame', the cabinet originally used to house the processing unit of such computers." Some discussions of the origin of the word "mainframe" are here, here, here, here, and here.
The phrase "main frame" in non-computer contexts has a very old but irrelevant history, describing many things that have a frame. For example, it appears in thousands of patents from the 1800s, including drills, saws, a meat cutter, a cider mill, printing presses, and corn planters. This shows that it was natural to use the phrase "main frame" when describing something constructed from frames. Telephony uses a Main distribution frame or "main frame" for wiring, going back to 1902. Some people claim that the computer use of "mainframe" is related to the telephony use, but I don't think they are related. In particular, a telephone main distribution frame looks nothing like a computer mainframe. Moreover, the computer use and the telephony use developed separately; if the computer use started in, say, Bell Labs, a connection would be more plausible.
IBM patents with "main frame" include a scale (1922), a card sorter (1927), a card duplicator (1929), and a card-based accounting machine (1930). IBM's incidental uses of "main frame" are probably unrelated to modern usage, but they are a reminder that punch card data processing started decades before the modern computer. ↩
It is unclear why the IBM 701 installation manual is dated August 27, 1952 but the drawing is dated 1953. I assume the drawing was updated after the manual was originally produced. ↩
This footnote will survey the construction techniques of some early computers; the key point is that building a computer on frames was not an obvious technique. ENIAC (1945), the famous early vacuum tube computer, was constructed from 40 panels forming three walls filling a room (ref, ref). EDVAC (1949) was built from large cabinets or panels (ref) while ORDVAC and CLADIC (1949) were built on racks (ref). One of the first commercial computers, UNIVAC 1 (1951), had a "Central Computer" organized as bays, divided into three sections, with tube "chassis" plugged in (ref ). The Raytheon computer (1951) and Moore School Automatic Computer (1952) (ref) were built from racks. The MONROBOT VI (1955) was described as constructed from the "conventional rack-panel-cabinet form" (ref). ↩
The size and construction of early computers often made it difficult to install or move them. The early computer ENIAC required 9 months to move from Philadelphia to the Aberdeen Proving Ground. For this move, the wall of the Moore School in Philadelphia had to be partially demolished so ENIAC's main panels could be removed. In 1959, moving the SWAC computer required disassembly of the computer and removing one wall of the building (ref). When moving the early computer JOHNNIAC to a different site, the builders discovered the computer was too big for the elevator. They had to raise the computer up the elevator shaft without the elevator (ref). This illustrates the benefits of building a computer from moveable frames. ↩
The IBM 701's main frame was called the Electronic Analytical Control Unit in external documentation. ↩
The 701 installation manual (1952) has a frame arrangement diagram showing the dimensions of the various frames, along with a drawing of the main frame, and power usage of the various frames. Service documentation (1953) refers to "main frame adjustments" (page 74). The 700 Series Data Processing Systems Component Circuits document (1955-1959) lists various types of frames in its abbreviation list (below)
When repairing an IBM 701, it was important to know which frame held which components, so "main frame" appeared throughout the engineering documents. For instance, in the schematics, each module was labeled with its location; "MF" stands for "main frame."
The "main frame" terminology was used in discussions with customers. For example, notes from a meeting with IBM (April 8, 1952) mention "E. S. [Electrostatic] Memory 15 feet from main frame" and list "main frame" as one of the seven items obtained for the $15,000/month rental cost. ↩
For more information on how the IBM 701 was designed to fit on elevators and through doorways, see Building IBM: Shaping an Industry and Technology page 170, and The Interface: IBM and the Transformation of Corporate Design page 69. This is also mentioned in "Engineering Description of the IBM Type 701 Computer", Proceedings of the IRE Oct 1953, page 1285. ↩
Many early systems used "central computer" to describe the main part of the computer, perhaps more commonly than "main frame." An early example is the "central computer" of the Elecom 125 (1954). The Digital Computer Newsletter (Apr 1955) used "central computer" several times to describe the processor of SEAC. The 1961 BRL report shows "central computer" being used by Univac II, Univac 1107, Univac File 0, DYSEAC and RCA Series 300. The MIT TX-2 Technical Manual (1961) uses "central computer" very frequently. The NAREC glossary (1962) defined "central computer. That part of a computer housed in the main frame." ↩
This footnote lists some other early computers that used the term "main frame." The October 1956 Digital Computer Newsletter mentions the "main frame" of the IBM NORC. Digital Computer Newsletter (Jan 1959) discusses using a RAMAC disk drive to reduce "main frame processing time." This document also mentions the IBM 709 "main frame." The IBM 704 documentation (1958) says "Each DC voltage is distributed to the main frame..." (IBM 736 reference manual) and "Check the air filters in each main frame unit and replace when dirty." (704 Central Processing Unit).
The July 1962 Digital Computer Newsletter discusses the LEO III computer: "It has been built on the modular principle with the main frame, individual blocks of storage, and input and output channels all physically separate." The article also mentions that the new computer is more compact with "a reduction of two cabinets for housing the main frame."
The IBM 7040 (1964) and IBM 7090 (1962) were constructed from multiple frames, including the processing unit called the "main frame."11 Machines in IBM's System/360 line (1964) were built from frames; some models had a main frame, power frame, wall frame, and so forth, while other models simply numbered the frames sequentially.12 ↩
The 1952 JOHNNIAC progress report is quoted in The History of the JOHNNIAC. This memorandum was dated August 8, 1952, so it is the earliest citation that I found. The June 1953 memorandum also used the term, stating, "The main frame is complete." ↩
A detailed description of IBM's frame-based computer packaging is in Standard Module System Component Circuits pages 6-9. This describes the SMS-based packaging used in the IBM 709x computers, the IBM 1401, and related systems as of 1960. ↩
IBM System/360 computers could have many frames, so they were usually given sequential numbers. The Model 85, for instance, had 12 frames for the processor and four megabytes of memory in 18 frames (at over 1000 pounds each). Some of the frames had descriptive names, though. The Model 40 had a main frame (CPU main frame, CPU frame), a main storage logic frame, a power supply frame, and a wall frame. The Model 50 had a CPU frame, power frame, and main storage frame. The Model 75 had a main frame (consisting of multiple physical frames), storage frames, channel frames, central processing frames, and a maintenance console frame. The compact Model 30 consisted of a single frame, so the documentation refers to the "frame", not the "main frame." For more information on frames in the System/360, see 360 Physical Planning. The Architecture of the IBM System/360 paper refers to the "main-frame hardware." ↩
A few more examples that discuss the minicomputer's mainframe, its physical box: A 1970 article discusses the mainframe of a minicomputer (as opposed to the peripherals) and contrasts minicomputers with large scale computers. A 1971 article on minicomputers discusses "minicomputer mainframes." Computerworld (Jan 28, 1970, p59) discusses minicomputer purchases: "The actual mainframe is not the major cost of the system to the user." Modern Data (1973) mentions minicomputer mainframes several times. ↩
DEC documents refer to the PDP-11 minicomputer as a mainframe. The PDP-11 Conventions manual (1970) defined: "Processor: A unit of a computing system that includes the circuits controlling the interpretation and execution of instructions. The processor does not include the Unibus, core memory, interface, or peripheral devices. The term 'main frame' is sometimes used but this term refers to all components (processor, memory, power supply) in the basic mounting box." In 1976, DEC published the PDP-11 Mainframe Troubleshooting Guide. The PDP-11 mainframe is also mentioned in Computerworld (1977). ↩
Test equipment manufacturers started using the term "main frame" (and later "mainframe") around 1962, to describe an oscilloscope or other test equipment that would accept plug-in modules. I suspect this is related to the use of "mainframe" to describe a computer's box, but it could be independent. Hewlett-Packard even used the term to describe a solderless breadboard, the 5035 Logic Lab. The Oxford English Dictionary (1989) used HP's 1974 ad for the Logic Lab as its earliest citation of mainframe as a single word. It appears that the OED confused this use of "mainframe" with the computer use.
In the 1980s, the use of "mainframe" to describe the box holding a microcomputer started to conflict with "mainframe" as a large computer. For example, Radio Electronics (October 1982), started using the short-lived term "micro-mainframe" instead of "mainframe" for a microcomputer's enclosure. By 1985, Byte magazine had largely switched to the modern usage of "mainframe." But even as late as 1987, a review of the Apple IIGC described one of the system's components as the '"mainframe" (i.e. the actual system box)'. ↩
Definitions of "central processing unit" disagreed as to whether storage was part of the CPU, part of the main frame, or something separate. This was largely a consequence of the physical construction of early computers. Smaller computers had memory in the same frame as the processor, while larger computers often had separate storage frames for memory. Other computers had some memory with the processor and some external. Thus, the "main frame" might or might not contain memory, and this ambiguity carried over to definitions of CPU. (In modern usage, the CPU consists of the arithmetic/logic unit (ALU) and control circuitry, but excludes memory.) ↩
Many definitions of mainframe or CPU mention "special register groups", an obscure feature specific to the Honeywell 800 computer (1959). (Processors have registers, special registers are common, and some processors have register groups, but only the Honeywell 800 had "special register groups.") However, computer dictionaries kept using this phrase for decades, even though it doesn't make sense for other computers. I wrote a blog post about special register groups here. ↩
This footnote provides more examples of "mainframe" being defined as the CPU. The Data Processing Equipment Encyclopedia (1961) had a similar definition: "Main Frame: The main part of the computer, i.e. the arithmetic or logic unit; the central processing unit." The 1967 IBM 360 operator's guide defined: "The main frame - the central processing unit and main storage." The Department of the Navy's ADP Glossary (1970): "Central processing unit: A unit of a computer that includes the circuits controlling the interpretation and execution of instructions. Synonymous with main frame." This was a popular definition, originally from the ISO, used by IBM (1979) among others. Funk & Wagnalls Dictionary of Data Processing Terms (1970) defined: "main frame: The basic or essential portion of an assembly of hardware, in particular, the central processing unit of a computer." The American National Standard Vocabulary for Information Processing (1970) defined: "central processing unit: A unit of a computer that includes the circuits controlling the interpretation and execution of instructions. Synonymous with main frame." ↩
Both the mainframe vs. peripheral definition and the mainframe as CPU definition made it unclear exactly what components of the computer were included in the mainframe. It's clear that the arithmetic-logic unit and the processor control circuitry were included, while I/O devices were excluded, but some components such as memory were in a gray area. It's also unclear if the power supply and I/O interfaces (channels) are part of the mainframe. These distinctions were ignored in almost all of the uses of "mainframe" that I saw.
An unusual definition in a Goddard Space Center document (1965, below) partitioned equipment into the "main frame" (the electronic equipment), "peripheral equipment" (electromechanical components such as the printer and tape), and "middle ground equipment" (the I/O interfaces). The "middle ground" terminology here appears to be unique. Also note that computers are partitioned into "super speed", "large-scale", "medium-scale", and "small-scale."
This footnote gives some examples of using peripherals to save the cost of mainframe time. IBM 650 documentation (1956) describes how "Data written on tape by the 650 can be processed by the main frame of the 700 series systems." Univac II Marketing Material (1957) discusses various ways of reducing "main frame time" by, for instance, printing from tape off-line. The USAF Guide for auditing automatic data processing systems (1961) discusses how these "off line" operations make the most efficient use of "the more expensive main frame time." ↩
Peripheral manufacturers were companies that built tape drives, printers, and other devices that could be connected to a mainframe built by IBM or another company. The basis for the peripheral industry was antitrust action against IBM that led to the 1956 Consent Decree. Among other things, the consent decree forced IBM to provide reasonable patent licensing, which allowed other firms to build "plug-compatible" peripherals. The introduction of the System/360 in 1964 produced a large market for peripherals and IBM's large profit margins left plenty of room for other companies. ↩
Computers and Automation, March 1965, categorized computers into five classes, from "Teeny systems" (such as the IBM 360/20) renting for $2000/month, through Small, Medium, and Large systems, up to "Family or Economy Size Systems" (such as the IBM 360/92) renting for $75,000 per month. ↩
The term "minicomputer" was supposedly invented by John Leng, head of DEC's England operations. In the 1960s, he sent back a sales report: "Here is the latest minicomputer activity in the land of miniskirts as I drive around in my Mini Minor", which led to the term becoming popular at DEC. This story is described in The Ultimate Entrepreneur: The Story of Ken Olsen and Digital Equipment Corporation (1988). I'd trust the story more if I could find a reference that wasn't 20 years after the fact. ↩
For instance, Computers and Automation (1971) discussed the role of the minicomputer as compared to "larger computers." A 1975 minicomputer report compared minicomputers to their "general-purpose cousins." ↩
This footnote provides more on the split between minicomputers and mainframes. In 1971, Modern Data Products, Systems, Services contained .".. will offer mainframe, minicomputer, and peripheral manufacturers a design, manufacturing, and production facility...." Standard & Poor's Industry Surveys (1972) mentions "mainframes, minicomputers, and IBM-compatible peripherals." Computerworld (1975) refers to "mainframe and minicomputer systems manufacturers."
The 1974 textbook "Information Systems: Technology, Economics, Applications" couldn't decide if mainframes were a part of the computer or a type of computer separate from minicomputers, saying: "Computer mainframes include the CPU and main memory, and in some usages of the term, the controllers, channels, and secondary storage and I/O devices such as tape drives, disks, terminals, card readers, printers, and so forth. However, the equipment for storage and I/O are usually called peripheral devices. Computer mainframes are usually thought of as medium to large scale, rather than mini-computers."
Studying U.S. Industrial Outlook reports provides another perspective over time. U.S. Industrial Outlook 1969 divides computers into small, medium-size, and large-scale. Mainframe manufacturers are in opposition to peripheral manufacturers. The same mainframe vs. peripherals opposition appears in U.S. Industrial Outlook 1970 and U.S. Industrial Outlook 1971. The 1971 report also discusses minicomputer manufacturers entering the "maxicomputer market."30 1973 mentions "large computers, minicomputers, and peripherals." U.S. Industrial Outlook 1976 states, "The distinction between mainframe computers, minis, micros, and also accounting machines and calculators should merge into a spectrum." By 1977, the market was separated into "general purpose mainframe computers", "minicomputers and small business computers" and "microprocessors."
Family Computing Magazine (1984) had a "Dictionary of Computer Terms Made Simple." It explained that "A Digital computer is either a "mainframe", a "mini", or a "micro." Forty years ago, large mainframes were the only size that a computer could be. They are still the largest size, and can handle more than 100,000,000 instructions per second. PER SECOND! [...] Mainframes are also called general-purpose computers." ↩
In 1974, Congress held antitrust hearings into IBM. The thousand-page report provides a detailed snapshot of the meanings of "mainframe" at the time. For instance, a market analysis report from IDC illustrates the difficulty of defining mainframes and minicomputers in this era (p4952). The "Mainframe Manufacturers" section splits the market into "general-purpose computers" and "dedicated application computers" including "all the so-called minicomputers." Although this section discusses minicomputers, the emphasis is on the manufacturers of traditional mainframes. A second "Plug-Compatible Manufacturers" section discusses companies that manufactured only peripherals. But there's also a separate "Minicomputers" section that focuses on minicomputers (along with microcomputers "which are simply microprocessor-based minicomputers"). My interpretation of this report is the terminology is in the process of moving from "mainframe vs. peripheral" to "mainframe vs. minicomputer." The statement from Research Shareholders Management (p5416) on the other hand discusses IBM and the five other mainframe companies; they classify minicomputer manufacturers separately. (p5425) p5426 mentions "mainframes, small business computers, industrial minicomputers, terminals, communications equipment, and minicomputers." Economist Ralph Miller mentions the central processing unit "(the so-called 'mainframe')" (p5621) and then contrasts independent peripheral manufacturers with mainframe manufacturers (p5622). The Computer Industry Alliance refers to mainframes and peripherals in multiple places, and "shifting the location of a controller from peripheral to mainframe", as well as "the central processing unit (mainframe)" p5099. On page 5290, "IBM on trial: Monopoly tends to corrupt", from Harper's (May 1974), mentions peripherals compatible with "IBM mainframe units—or, as they are called, central processing computers." ↩
The influential business newsletter EDP provides an interesting view on the struggle to separate the minicomputer market from larger computers. Through 1968, they included minicomputers in the "general-purpose computer" category. But in 1969, they split "general-purpose computers" into "Group A, General Purpose Digital Computers" and "Group B, Dedicated Application Digital Computers." These categories roughly corresponded to larger computers and minicomputers, on the (dubious) assumption that minicomputers were used for a "dedicated application." The important thing to note is that in 1969 they did not use the term "mainframe" for the first category, even though with the modern definition it's the obvious term to use. At the time, EDP used "mainframe manufacturer" or "mainframer"31 to refer to companies that manufactured computers (including minicomputers), as opposed to manufacturers of peripherals. In 1972, EDP first mentioned mainframes and minicomputers as distinct types. In 1973, "microcomputer" was added to the categories. As the 1970s progressed, the separation between minicomputers and mainframes became common. However, the transition was not completely smooth; 1973 included a reference to "mainframe shipments (including minicomputers)."
To specific, the EDP Industry Report (Nov. 28, 1969) gave the following definitions of the two groups of computers:
Group A—General Purpose Digital Computers: These comprise the bulk of the computers that have been listed in the Census previously. They are character or byte oriented except in the case of the large-scale scientific machines, which have 36, 48, or 60-bit words. The predominant portion (60% to 80%) of these computers is rented, usually for $2,000 a month or more. Higher level languages such as Fortran, Cobol, or PL/1 are the primary means by which users program these computers.
Group B—Dedicated Application Digital Computers: This group of computers includes the "mini's" (purchase price below $25,000), the "midi's" ($25,000 to $50,000), and certain larger systems usually designed or used for one dedicated application such as process control, data acquisition, etc. The characteristics of this group are that the computers are usually word oriented (8, 12, 16, or 24-bits per word), the predominant number (70% to 100%) are purchased, and assembly language (at times Fortran) is the predominant means of programming. This type of computer is often sold to an original equipment manufacturer (OEM) for further system integration and resale to the final user.
These definitions strike me as rather arbitrary. ↩
In 1981 Computerworld had articles trying to clarify the distinctions between microcomputers, minicomputers, superminicomputers, and mainframes, as the systems started to overlay. One article, Distinction Helpful for Minis, Mainframes said that minicomputers were generally interactive, while mainframes made good batch machines and network hosts. Microcomputers had up to 512 KB of memory, minis were 16-bit machines with 512 KB to 4 MB of memory, costing up to $100,000. Superminis were 16- to 32-bit machines with 4 MB to 8 MB of memory, costing up to $200,000 but with less memory bandwidth than mainframes. Finally, mainframes were 32-bit machines with more than 8 MB of memory, costing over $200,000. Another article Micro, Mini, or Mainframe? Confusion persists described a microcomputer as using an 8-bit architecture and having fewer peripherals, while a minicomputer has a 16-bit architecture and 48 KB to 1 MB of memory. ↩
The miniskirt in the mid-1960s was shortly followed by the midiskirt and maxiskirt. These terms led to the parallel construction of the terms minicomputer, midicomputer, and maxicomputer.
The New York Times had a long article Maxi Computers Face Mini Conflict (April 5, 1970) explicitly making the parallel: "Mini vs. Maxi, the reigning issue in the glamorous world of fashion, is strangely enough also a major point of contention in the definitely unsexy realm of computers."
Although midicomputer and maxicomputer terminology didn't catch on the way minicomputer did, they still had significant use (example, midicomputer examples, maxicomputer examples).
The miniskirt/minicomputer parallel was done with varying degrees of sexism. One example is Electronic Design News (1969): "A minicomputer. Like the miniskirt, the small general-purpose computer presents the same basic commodity in a more appealing way." ↩
Linguistically, one indication that a new word has become integrated in the language is when it can be extended to form additional new words. One example is the formation of "mainframers", referring to companies that build mainframes. This word was moderately popular in the 1970s to 1990s. It was even used by the Department of Justice in their 1975 action against IBM where they described the companies in the systems market as the "mainframe companies" or "mainframers." The word is still used today, but usually refers to people with mainframe skills. Other linguistic extensions of "mainframe" include mainframing, unmainframe, mainframed, nonmainframe, and postmainframe. ↩
More examples of the split between microcomputers and mainframes: Softwide Magazine (1978) describes "BASIC versions for micro, mini and mainframe computers." MSC, a disk system manufacturer, had drives "used with many microcomputer, minicomputer, and mainframe processor types" (1980). ↩
Some examples of computer dictionaries referring to mainframes as a size category: Illustrated Dictionary of Microcomputer Terminology (1978) defines "mainframe" as "(1) The heart of a computer system, which includes the CPU and ALU. (2) A large computer, as opposed to a mini or micro." A Dictionary of Minicomputing and Microcomputing (1982) includes the definition of "mainframe" as "A high-speed computer that is larger, faster, and more expensive than the high-end minicomputers. The boundary between a small mainframe and a large mini is fuzzy indeed." The National Bureau of Standards Future Information Technology (1984) defined: "Mainframe is a term used to designate a medium and large scale CPU." The New American Computer Dictionary (1985) defined "mainframe" as "(1) Specifically, the rack(s) holding the central processing unit and the memory of a large computer. (2) More generally, any large computer. 'We have two mainframes and several minis.'" The 1990 ANSI Dictionary for Information Systems (ANSI X3.172-1990) defined: mainframe. A large computer, usually one to which other computers are connected in order to share its resources and computing power. Microsoft Press Computer Dictionary (1991) defined "mainframe computer" as "A high-level computer designed for the most intensive computational tasks. Mainframe computers are often shared by multiple users connected to the computer via terminals." ISO 2382 (1993) defines a mainframe as "a computer, usually in a computer center, with extensive capabilities and resources to which other computers may be connected so that they can share facilities."
The Microsoft Computer Dictionary (2002) had an amusingly critical definition of mainframe: "A type of large computer system (in the past often water-cooled), the primary data processing resource for many large businesses and organizations. Some mainframe operating systems and solutions are over 40 years old and have the capacity to store year values only as two digits." ↩
IBM's 1962 book Planning a Computer System (1962) describes how the Stretch computer's circuitry was assembled into frames, with the CPU consisting of 18 frames. The picture below shows how a "frame" was, in fact, constructed from a metal frame.
The term "general-purpose computer" is probably worthy of investigation since it was used in a variety of ways. It is one of those phrases that seems obvious until you think about it more closely. On the one hand, a computer such as the Apollo Guidance Computer can be considered general purpose because it runs a variety of programs, even though the computer was designed for one specific mission. On the other hand, minicomputers were often contrasted with "general-purpose computers" because customers would buy a minicomputer for a specific application, unlike a mainframe which would be used for a variety of applications. ↩
The n-gram graph is from the Google Books Ngram Viewer. The curves on the graph should be taken with a grain of salt. First, the usage of words in published books is likely to lag behind "real world" usage. Second, the number of usages in the data set is small, especially at the beginning. Nonetheless, the n-gram graph generally agrees with what I've seen looking at documents directly. ↩
More examples of "mainframe" in want ads: A 1966 ad from Western Union in The Arizona Republic looking for experience "in a systems engineering capacity dealing with both mainframe and peripherals." A 1968 ad in The Minneapolis Star for an engineer with knowledge of "mainframe and peripheral hardware." A 1968 ad from SDS in The Los Angeles Times for an engineer to design "circuits for computer mainframes and peripheral equipment." A 1968 ad in Fort Lauderdale News for "Computer mainframe and peripheral logic design." A 1972 ad in The Los Angeles Times saying "Mainframe or peripheral [experience] highly desired." In most of these ads, the mainframe was in contrast to the peripherals. ↩
A related factor is the development of remote connections from a microcomputer to a mainframe in the 1980s. This led to the need for a word to describe the remote computer, rather than saying "I connected my home computer to the other computer." See the many books and articles on connecting "micro to mainframe." ↩
To see how the prototypical meaning of "computer" changed in the 1980s, I examined the "Computer" article in encyclopedias from that time. The 1980 Concise Encyclopedia of the Sciences discusses a large system with punched-card input. In 1980, the World Book article focused on mainframe systems, starting with a photo of an IBM System/360 Model 40 mainframe. But in the 1981 supplement and the 1984 encyclopedia, the World Book article opened with a handheld computer game, a desktop computer, and a "large-scale computer." The article described microcomputers, minicomputers, and mainframes. Funk & Wagnalls Encyclopedia (1983) was in the middle of the transition; the article focused on large computers and had photos of IBM machines, but mentioned that future growth is expected in microcomputers. By 1994, the World Book article's main focus was the personal computer, although the mainframe still had a few paragraphs and a photo. This is evidence that the prototypical meaning of "computer" underwent a dramatic shift in the early 1980s from a mainframe to a balance between small and large computers, and then to the personal computer. ↩
Intel released the powerful Pentium processor in 1993, establishing a long-running brand of processors. Earlier, I wrote about the ROM in the Pentium's floating point unit that holds constants such as π. In this post, I'll look at some interesting circuits associated with this ROM. In particular, the circuitry is implemented in BiCMOS, a process that combines bipolar transistors with standard CMOS logic.
The photo below shows the Pentium's thumbnail-sized silicon die under a microscope. I've labeled the main functional blocks; the floating point unit is in the lower right with the constant ROM highlighted at the bottom. The various parts of the floating point unit form horizontal stripes. Data buses run vertically through the floating point unit, moving values around the unit.
The diagram below shows how the circuitry in this post forms part of the Pentium. Zooming in to the bottom of the chip shows the constant ROM, holding 86-bit words: at the left, the exponent section provides 18 bits. At the right, the wider significand section provides 68 bits. Below that, the diagram zooms in on the subject of this article: one of the 86 identical multiplexer/driver circuits that provides the output from the ROM. As you can see, this circuit is a microscopic speck in the chip.
In this section, I'll show how the Pentium is constructed from layers. The bottom layer of the chip consists of transistors fabricated on the silicon die. Regions of silicon are doped with impurities to change the electrical properties; these regions appear pinkish in the photo below, compared to the grayish undoped silicon. Thin polysilicon wiring is formed on top of the silicon. Where a polysilicon line crosses doped silicon, a transistor is formed; the polysilicon creates the transistor's gate. Most of these transistors are NMOS and PMOS transistors, but there is a bipolar transistor near the upper right, the large box-like structure. The dark circles are contacts, regions where the metal layer above is connected to the polysilicon or silicon to wire the circuits together.
The Pentium has three layers of metal wiring. The photo below shows the bottom layer, called M1. For the most part, this layer of metal connects the transistors into various circuits, providing wiring over a short distance. The photos in this section show the same region of the chip, so you can match up features between the photos. For instance, the contacts below (black circles) match the black circles above, showing how this metal layer connects to the silicon and polysilicon circuits. You can see some of the silicon and polysilicon in this image, but most of it is hidden by the metal.
The M2 metal layer (below) sits above the M1 wiring. In this part of the chip, the M2 wires are horizontal. The thicker lines are power and ground. (Because they are thicker, they have lower resistance and can provide the necessary current to the underlying circuitry.) The thinner lines are control signals. The floating point unit is structured so functional blocks are horizontal, while data is transmitted vertically. Thus, a horizontal wire can supply a control signal to all the bits in a functional block.
The M3 layer is the top metal layer in the Pentium. It is thicker, so it is better suited for the chip's main power and ground lines as well as long-distance bus wiring. In the photo below, the wide line on the left provides power, while the wide line on the right provides ground. The power and ground are distributed through wiring in the M2 and M1 layers until they are connected to the underlying transistors. At the top of the photo, vertical bus lines are visible; these extend for long distances through the floating point unit. Notice the slightly longer line, fourth from the right. This line provides one bit of data from the ROM, provided by the circuitry described below. The dot near the bottom is a via, connecting this line to a short wire in M2, connected to a wire in M1, connected to the silicon of the output transistors.
The simplified schematic below shows the circuit that I reverse-engineered. This circuit is repeated 86 times, once for each bit in the ROM's word. You might expect the ROM to provide a single 86-bit word. However, to make the layout work better, the ROM provides eight words in parallel. Thus, the circuitry must select one of the eight words with a multiplexer. In particular, each of the 86 circuits has an 8-to-1 multiplexer to select one bit out of the eight. This bit is then stored in a latch. Finally, a high-current driver amplifies the signal so it can be sent through a bus, traveling to a destination halfway across the floating point unit.
I'll provide a quick review of MOS transistors before I explain the circuitry in detail. CMOS circuitry uses two types of transistors—PMOS and NMOS—which are similar but also opposites. A PMOS transistor is turned on by a low signal on the gate, while an NMOS transistor is turned on by a high signal on the gate; the PMOS symbol has an inversion bubble on the gate. A PMOS transistor works best when pulling its output high, while an NMOS transistor works best when pulling its output low. CMOS circuitry normally uses the two types of MOS transistors in a Complementary fashion to implement logic gates, working together. What makes the circuits below interesting is that they often use NMOS and PMOS transistors independently.
The detailed schematic below shows the circuitry at the transistor and inverter level. I'll go through each of the components in the remainder of this post.
The ROM is constructed as a grid: at each grid point, the ROM can have a transistor for a 0 bit, or no transistor for a 1 bit. Thus, the data is represented by the transistor pattern. The ROM holds 304 constants so there are 304 potential transistors associated with each bit of the output word. These transistors are organized in a 38×8 grid. To select a word from the ROM, a select line activates one group of eight potential transistors. Each transistor is connected to ground, so the transistor (if present) will pull the associated line low, for a 0 bit. Note that the ROM itself consists of only NMOS transistors, making it half the size of a truly CMOS implementation. For more information on the structure and contents of the ROM, see my earlier article.
A ROM transistor can pull a line low for a 0 bit, but how does the line get pulled high for a 1 bit? This is accomplished by a precharge transistor on each line. Before a read from the ROM, the precharge transistors are all activated, pulling the lines high. If a ROM transistor is present on the line, the line will next be pulled low, but otherwise it will remain high due to the capacitance on the line.
Next, the multiplexer above selects one of the 8 lines, depending on which word is being accessed. The multiplexer consists of eight transistors. One transistor is activated by a select line, allowing the ROM's signal to pass through. The other seven transistors are in the off state, blocking those ROM signals. Thus, the multiplexer selects one of the 8 bits from the ROM.
The circuit below is the "keeper." As explained above, each ROM line is charged high before reading the ROM. However, this charge can fade away. The job of the keeper is to keep the multiplexer's output high until it is pulled low. This is implemented by an inverter connected to a PMOS transistor. If the signal on the line is high, the PMOS transistor will turn on, pulling the line high. (Note that a PMOS transistor is turned on by a low signal, thus the inverter.) If the ROM pulls the line low, the transistor will turn off and stop pulling the line high. This transistor is very weak, so it is easily overpowered by the signal from the ROM. The transistor on the left ensures that the line is high at the start of the cycle.
The diagram below shows the transistors for the keeper. The two transistors on the left implement a standard CMOS inverter. On the right, note the weak transistor that holds the line high. You might notice that the weak transistor looks larger and wonder why that makes the transistor weak rather than strong. The explanation is that the transistor is large in the "wrong" dimension. The current capacity of an MOS transistor is proportional to the width/length ratio of its gate. (Width is usually the long dimension and length is usually the skinny dimension.) The weak transistor's length is much larger than the other transistors, so the W/L ratio is smaller and the transistor is weaker. (You can think of the transistor's gate as a bridge between its two sides. A wide bridge with many lanes lets lots of traffic through. However, a long, single-lane bridge will slow down the traffic.)
Next, we come to the latch, which remembers the value read from the ROM. This latch will read its input when the load signal is high. When the load signal goes low, the latch will hold its value. Conceptually, the latch is implemented with the circuit below. A multiplexer selects the lower input when the load signal is active, passing the latch input through to the (inverted) output. But when the load signal goes low, the multiplexer will select the top input, which is feedback of the value in the latch. This signal will cycle through the inverters and the multiplexer, holding the value until a new value is loaded. The inverters are required because the multiplexer itself doesn't provide any amplification; the signal would rapidly die out if not amplified by the inverters.
The multiplexer is implemented with two CMOS switches, one to select each multiplexer input. Each switch is a pair of PMOS and NMOS transistors that turn on together, allowing a signal to pass through. (See the bottom two transistors below.)1 The upper circuit is trickier. Conceptually, it is an inverter feeding into the multiplexer's CMOS switch. However, the order is switched so the switch feeds into the inverter. The result is not-exactly-a-switch and not-exactly-an-inverter, but the result is the same. You can also view it as an inverter with power and ground that gets cut off when not selected. I suspect this implementation uses slightly less power than the straightforward implementation.
The most unusual circuit is the BiCMOS driver. By adding a few extra processing steps to the regular CMOS manufacturing process, bipolar (NPN and PNP) transistors can be created. The Pentium extensively used BiCMOS circuits since they reduced signal delays by up to 35%. Intel also used BiCMOS for the Pentium Pro, Pentium II, Pentium III, and Xeon processors. However, as chip voltages dropped, the benefit from bipolar transistors dropped too and BiCMOS was eventually abandoned.
In the Pentium, BiCMOS drivers are used when signals must travel a long distance across the chip. (In this case, the ROM output travels about halfway up the floating point unit.) These long wires have a lot of capacitance so a high-current driver circuit is needed and the NPN transistor provides extra "oomph."
The diagram below shows how the driver is implemented. The NPN transistor is the large boxy structure in the upper right. When the base (B) is pulled high, current flows from the collector (C), pulling the emitter (E) high and thus rapidly pulling the output high. The remainder of the circuit consists of three inverters, each composed of PMOS and NMOS transistors. When a polysilicon line crosses doped silicon, it creates a transistor gate, so each crossing corresponds to a transistor. The inverters use multiple transistors in parallel to provide more current; the transistor sources and/or drains overlap to make the circuitry more compact.
One interesting thing about this circuit is that each inverter is carefully designed to provide the desired current, with a different current for a high output versus a low output. The first inverter (purple boxes) has two PMOS transistors and two NMOS transistors, so it is a regular inverter, balanced for high and low outputs. (This inverter is conceptually part of the latch.) The second inverter (yellow boxes) has three large PMOS transistors and one smaller NMOS transistor, so it has more ability to pull the output high than low. This transistor turns on the NPN transistor by providing a high signal to the base, so it needs more current in the high state. The third inverter (green boxes) has one weak PMOS transistor and seven NMOS transistors, so it can pull its output low strongly, but can barely pull its output high. This transistor pulls the ROM output line low, so it needs enough current to drive the entire bus line. But this transistor doesn't need to pull the output high—that's the job of the NPN transistor—so the PMOS transistor can be weak. The construction of the weak transistor is similar to the keeper's weak transistor; its gate length is much larger than the other transistors, so it provides less current.
The diagram below shows how the functional blocks are arranged in the complete circuit, from the ROM at the bottom to the output at the top. The floating point unit is constructed with a constant width for each bit—38.5 µm—so the circuitry is designed to fit into this width. The layout of this circuitry was hand-optimized to fit as tightly as possible, In comparison, much of the Pentium's circuitry was arranged by software using a standard-cell approach, which is much easier to design but not as dense. Since each bit in the floating point unit is repeated many times, hand-optimization paid off here.
This circuit contains 47 transistors. Since it is duplicated once for each bit, it has 4042 transistors in total, a tiny fraction of the Pentium's 3.1 million transistors. In comparison, the MOS 6502 processor has about 3500-4500 transistors, depending on how you count. In other words, the circuit to select a word from the Pentium's ROM is about as complex as the entire 6502 processor. This illustrates the dramatic growth in processor complexity described by Moore's law.
I plan to write more about the Pentium so follow me on Bluesky (@righto.com) or RSS for updates. (I'm no longer on Twitter.) You might enjoy reading about the Pentium Navajo rug.
The 8-to-1 multiplexer and the latch's multiplexer use different switch implementations: the first is built from NMOS transistors while the second is built from paired PMOS and NMOS transistors. The reason is that NMOS transistors are better at pulling signals low, while PMOS transistors are better at pulling signals high. Combining the transistors creates a switch that passes low and high signals efficiently, which is useful in the latch. The 8-to-1 multiplexer, however, only needs to pull signals low (due to the precharging), so the NMOS-only multiplexer works in this role. (Note that early NMOS processors like the 6502 and 8086 built multiplexers and pass-transistor logic out of solely NMOS. This illustrates that you can use NMOS-only switches with both logic levels, but performance is better if you add PMOS transistors.) ↩
Addition is harder than you'd expect, at least for a computer. Computers use multiple types of adder circuits with different tradeoffs of size versus speed. In this article, I reverse-engineer an 8-bit adder in the Pentium's floating point unit. This adder turns out to be a carry-lookahead adder, in particular, a type known as "Kogge-Stone."1 In this article, I'll explain how a carry-lookahead adder works and I'll show how the Pentium implemented it. Warning: lots of Boolean logic ahead.
The die photo above shows the main functional units of the Pentium. The adder, in the lower right, is a small component of the floating point unit. It is not a general-purpose adder, but is used only for determining quotient digits during division. It played a role in the famous Pentium FDIV division bug, which I wrote about here.
The photo below shows the carry-lookahead adder used by the divider. The adder itself consists of the circuitry highlighted in red. At the top, logic gates compute signals in parallel for each of the 8 pairs of inputs: partial sum, carry generate, and carry propagate. Next, the complex carry-lookahead logic determines in parallel if there will be a carry at each position. Finally, XOR gates apply the carry to each bit. Note that the sum/generate/propagate circuitry consists of 8 repeated blocks, and the same with the carry XOR circuitry. The carry lookahead circuitry, however, doesn't have any visible structure since it is different for each bit.2
The large amount of circuitry in the middle is used for testing; see the footnote.3 At the bottom, the drivers amplify control signals for various parts of the circuit.
The problem with addition is that carries make addition slow. Consider calculating 99999+1 by hand. You'll start with 9+1=10, then carry the one, generating another carry, which generates another carry, and so forth, until you go through all the digits. Computer addition has the same problem: If you're adding two numbers, the low-order bits can generate a carry that then propagates through all the bits. An adder that works this way—known as a ripple carry adder—will be slow because the carry has to ripple through all the bits. As a result, CPUs use special circuits to make addition faster.
One solution is the carry-lookahead adder. In this adder, all the carry bits are computed in parallel, before computing the sums. Then, the sum bits can be computed in parallel, using the carry bits. As a result, the addition can be completed quickly, without waiting for the carries to ripple through the entire sum.
It may seem impossible to compute the carries without computing the sum first, but there's a way to do it.
For each bit position, you determine signals called "carry generate" and "carry propagate".
These signals can then be used to determine all the carries in parallel.
The generate signal indicates that the position generates a carry. For instance, if you add binary
1xx and 1xx (where x is an arbitrary bit), a carry will be generated from the top bit,
regardless of the unspecified bits.
On the other hand, adding 0xx and 0xx will never produce a carry.
Thus, the generate signal is produced for the first case but not the second.
But what about 1xx plus 0xx? We might get a carry, for instance, 111+001, but we might not get a carry,
for instance, 101+001. In this "maybe" case, we set the carry propagate signal, indicating that a carry into the
position will get propagated out of the position. For example, if there is a carry out of
the middle position, 1xx+0xx will have a carry from the top bit. But if there is no carry out of the middle position, then
there will not be a carry from the top bit. In other words, the propagate signal indicates that a carry into the top bit will be propagated out of the top
bit.
To summarize, adding 1+1 will generate a carry. Adding 0+1 or 1+0 will propagate a
carry.
Thus, the generate signal is formed at each position by Gn = An·Bn, where A and B are the inputs.
The propagate signal is Pn = An+Bn,
the logical-OR of the inputs.4
Now that the propagate and generate signals are defined, they can be used to compute the carry Cn at
each bit position:
C1 = G0: a carry into bit 1 occurs if a carry is generated from bit 0.
C2 = G1 + G0P1: A carry into bit 2 occur if bit 1 generates a carry or bit 1 propagates a carry from bit 0.
C3 = G2 + G1P2 + G0P1P2: A carry into bit 3 occurs if bit 2 generates a carry, or bit 2 propagates a carry generated from bit 1, or bits 2 and 1 propagate a carry generated from bit 0.
C4 = G3 + G2P3 + G1P2P3 + G0P1P2P3: A carry into bit 4 occurs if a carry is generated from bit 3, 2, 1, or 0 along with the necessary propagate signals.
... and so forth, getting more complicated with each bit ...
The important thing about these equations is that they can be computed in parallel, without waiting for a carry to ripple through each position. Once each carry is computed, the sum bits can be computed in parallel: Sn = An ⊕ Bn ⊕ Cn. In other words, the two input bits and the computed carry are combined with exclusive-or.
The straightforward way to implement carry lookahead is to directly implement the equations above. However, this approach requires a lot of circuitry due to the complicated equations. Moreover, it needs gates with many inputs, which are slow for electrical reasons.5
The Pentium's adder implements the carry lookahead in a different way, called the "parallel prefix adder."7 The idea is to produce the propagate and generate signals across ranges of bits, not just single bits as before. For instance, the propagate signal P32 indicates that a carry in to bit 2 would be propagated out of bit 3. And G30 indicates that bits 3 to 0 generate a carry out of bit 3.
Using some mathematical tricks,6 you can take the P and G values for two smaller ranges and merge them into the P and G values for the combined range. For instance, you can start with the P and G values for bits 0 and 1, and produce P10 and G10. These could be merged with P32 and G32 to produce P30 and G30, indicating if a carry is propagated across bits 3-0 or generated by bits 3-0. Note that Gn0 is the carry-lookahead value we need for bit n, so producing these G values gives the results that we need from the carry-lookahead implementation.
This merging process is more efficient than the "brute force" implementation of the carry-lookahead logic since logic subexpressions can be reused. This merging process can be implemented in many ways, including Kogge-Stone, Brent-Kung, and Ladner-Fischer. The different algorithms have different tradeoffs of performance versus circuit area. In the next section, I'll show how the Pentium implements the Kogge-Stone algorithm.
The Pentium's adder is implemented with four layers of circuitry. The first layer produces the propagate and generate signals (P and G) for each bit, along with a partial sum (the sum without any carries). The second layer merges pairs of neighboring P and G values, producing, for instance G65 and P21. The third layer generates the carry-lookahead bits by merging previous P and G values. This layer is complicated because it has different circuitry for each bit. Finally, the fourth layer applies the carry bits to the partial sum, producing the final arithmetic sum.
Here is the schematic of the adder, from my reverse engineering. The circuit in the upper left is repeated 8 times to produce the propagate, generate, and partial sum for each bit. This corresponds to the first layer of logic. At the left are the circuits to merge the generate and propagate signals across pairs of bits. These circuits are the second layer of logic.
The circuitry at the right is the interesting part—it computes the carries in parallel and then computes the final sum bits using XOR. This corresponds to the third and fourth layers of circuitry respectively. The circuitry gets more complicated going from bottom to top as the bit position increases.
The diagram below is the standard diagram that illustrates how a Kogge-Stone adder works. It's rather abstract, but I'll try to explain it. The diagram shows how the P and G signals are merged to produce each output at the bottom. Each line coresponds to both the P and the G signal. Each square box generates the P and G signals for that bit. (Confusingly, the vertical and diagonal lines have the same meaning, indicating inputs going into a diamond and outputs coming out of a diamond.) Each diamond combines two ranges of P and G signals to generate new P and G signals for the combined range. Thus, the signals cover wider ranges as they progress downward, ending with the Gn0 signals that are the outputs.
It may be easier to understand the diagram by starting with the outputs. I've highlighted two circuits: The purple circuit computes the carry into bit 3 (out of bit 2), while the green circuit computes the carry into bit 7 (out of bit 6). Following the purple output upward, note that it forms a tree reaching bits 2, 1, and 0, so it generates the carry based on these bits, as desired. In more detail, the upper purple diamond combines the P and G signals for bits 2 and 1, generating P21 and G21. The lower purple diamond merges in P0 and G0 to create P20 and G20. Signal G20 indicates of bits 2 through 0 generate a carry; this is the desired carry value into bit 3.
Now, look at the green output and see how it forms a tree going upward, combining bits 6 through 0. Notice how it takes advantage of the purple carry output, reducing the circuitry required. It also uses P65, P43, and the corresponding G signals. Comparing with the earlier schematic shows how the diagram corresponds to the schematic, but abstracts out the details of the gates.
Comparing the diagram to the schematic, each square box corresponds to to the circuit in the upper left of the schematic that generates P and G, the first layer of circuitry. The first row of diamonds corresponds to the pairwise combination circuitry on the left of the schematic, the second layer of circuitry. The remaining diamonds correspond to the circuitry on the right of the schematic, with each column corresponding to a bit, the third layer of circuitry. (The diagram ignores the final XOR step, the fourth layer of circuitry.)
Next, I'll show how the diagram above, the logic equations, and the schematic are related. The diagram below shows the logic equation for C7 and how it is implemented with gates; this corresponds to the green diamonds above. The gates on the left below computes G63; this corresponds to the middle green diamond on the left. The next gate below computes P63 from P65 and P43; this corresponds to the same green diamond. The last gates mix in C3 (the purple line above); this corresponds to the bottom green diamond. As you can see, the diamonds abstract away the complexity of the gates. Finally, the colored boxes below show how the gate inputs map onto the logic equation. Each input corresponds to multiple terms in the equation (6 inputs replace 28 terms), showing how this approach reduces the circuitry required.
There are alternatives to the Kogge-Stone adder. For example, a Brent-Kung adder (below) uses a different arrangement with fewer diamonds but more layers. Thus, a Brent-Kung adder uses less circuitry but is slower. (You can follow each output upward to verify that the tree reaches the correct inputs.)
The photo below shows the adder circuitry. I've removed the top two layers of metal, leaving the bottom layer of metal. Underneath the metal, polysilicon wiring and doped silicon regions are barely visible; they form the transistors. At the top are eight blocks of gates to generate the partial sum, generate, and propagate signals for each bit. (This corresponds to the first layer of circuitry as described earlier.) In the middle is the carry lookahead circuitry. It is irregular since each bit has different circuitry. (This corresponds to the second and third layers of circuitry, jumbled together.) At the bottom, eight XOR gates combine the carry lookahead output with the partial sum to produce the adder's output. (This corresponds to the fourth layer of circuitry.)
The Pentium uses many adders for different purposes: in the integer unit, in the floating point unit, and for address calculation, among others. Floating-point division is known to use a carry-save adder to hold the partial remainder at each step; see my post on the Pentium FDIV division bug for details. I don't know what types of adders are used in other parts of the chip, but maybe I'll reverse-engineer some of them. Follow me on Bluesky (@righto.com) or RSS for updates. (I'm no longer on Twitter.)
Strangely, the original paper by Kogge and Stone had nothing to do with addition and carries. Their 1973 paper was titled, "A Parallel Algorithm for the Efficient Solution of a General Class of Recurrence Equations." It described how to solve recurrence problems on parallel computers, in particular the massively parallel ILLIAC IV. As far as I can tell, it wasn't until 1987 that their algorithm was applied to carry lookahead, in Fast Area-Efficient VLSI Adders. ↩
I'm a bit puzzled why the circuit uses an 8-bit carry-lookahead adder since only 7 bits are used. Moreover, the carry-out is unused. However, the adder's bottom output bit is not connected to anything. Perhaps the 8-bit adder was a standard logic block at Intel and was used as-is. ↩
I probably won't make a separate blog post on the testing circuitry, so I'll put details in this footnote. Half of the circuitry in the adder block is used to test the lookup table. The reason is that a chip such as the Pentium is very difficult to test: if one out of 3.1 million transistors goes bad, how do you detect it? For a simple processor like the 8080, you can run through the instruction set and be fairly confident that any problem would turn up. But with a complex chip, it is almost impossible to come up with an instruction sequence that would test every bit of the microcode ROM, every bit of the cache, and so forth. Starting with the 386, Intel added circuitry to the processor solely to make testing easier; about 2.7% of the transistors in the 386 were for testing.
To test a ROM inside the processor, Intel added circuitry to scan the entire ROM and checksum its contents. Specifically, a pseudo-random number generator runs through each address, while another circuit computes a checksum of the ROM output, forming a "signature" word. At the end, if the signature word has the right value, the ROM is almost certainly correct. But if there is even a single bit error, the checksum will be wrong and the chip will be rejected. The pseudo-random numbers and the checksum are both implemented with linear feedback shift registers (LFSR), a shift register along with a few XOR gates to feed the output back to the input. For more information on testing circuitry in the 386, see Design and Test of the 80386, written by Pat Gelsinger, who became Intel's CEO years later. Even with the test circuitry, 48% of the transistor sites in the 386 were untested. The instruction-level test suite to test the remaining circuitry took almost 800,000 clock cycles to run. The overhead of the test circuitry was about 10% more transistors in the blocks that were tested.
In the Pentium, the circuitry to test the lookup table PLA is just below the 7-bit adder. An 11-bit LFSR creates the 11-bit input value to the lookup table. A 13-bit LFSR hashes the two-bit quotient result from the PLA, forming a 13-bit checksum. The checksum is fed serially to test circuitry elsewhere in the chip, where it is merged with other test data and written to a register. If the register is 0 at the end, all the tests pass. In particular, if the checksum is correct, you can be 99.99% sure that the lookup table is operating as expected. The ironic thing is that this test circuit was useless for the FDIV bug: it ensured that the lookup table held the intended values, but the intended values were wrong.
Why did Intel generate test addresses with a pseudo-random sequence instead of a sequential counter? It turns out that a linear feedback shift register (LFSR) is slightly more compact than a counter. This LFSR trick was also used in a touch-tone chip and the program counter of the Texas Instruments TMS 1000 microcontroller (1974). In the TMS 1000, the program counter steps through the program pseudo-randomly rather than sequentially. The program is shuffled appropriately in the ROM to counteract the sequence, so the program executes as expected and a few transistors are saved.
The bits 1+1 will set generate, but should propagate be set too?
It doesn't make a difference as far as the equations. This adder sets propagate for 1+1 but some
other adders do not.
The answer depends on if you use an inclusive-or or exclusive-or gate
to produce the propagate signal. ↩
One solution is to implement the carry-lookahead circuit in blocks of four. This can be scaled up with a second level of carry-lookahead to provide the carry lookahead across each group of four blocks. A third level can provide carry lookahead for groups of four second-level blocks, and so forth. This approach requires O(log(N)) levels for N-bit addition. This approach is used by the venerable 74181 ALU, a chip used by many minicomputers in the 1970s; I reverse-engineered the 74181 here. The 74182 chip provides carry lookahead for the higher levels. ↩
I won't go into the mathematics of merging P and G signals; see, for example, Adder Circuits, Adders, or Carry Lookahead Adders for additional details. The important factor is that the carry merge operator is associative (actually a monoid), so the sub-ranges can be merged in any order. This flexibility is what allows different algorithms with different tradeoffs. ↩
The idea behind a prefix adder is that we want to see if there is a carry out of bit 0, bits 0-1, bits 0-2, bits 0-3, 0-4, and so forth. These are all the prefixes of the word. Since the prefixes are computed in parallel, it's called a parallel prefix adder. ↩
The cargo cult metaphor is commonly used by programmers. This metaphor was popularized by Richard Feynman's "cargo cult science" talk with a vivid description of South Seas cargo cults. However, this metaphor has three major problems. First, the pop-culture depiction of cargo cults is inaccurate and fictionalized, as I'll show. Second, the metaphor is overused and has contradictory meanings making it a lazy insult. Finally, cargo cults are portrayed as an amusing story of native misunderstanding but the background is much darker: cargo cults are a reaction to decades of oppression of Melanesian islanders and the destruction of their culture. For these reasons, the cargo cult metaphor is best avoided.
In this post, I'll describe some cargo cults from 1919 to the present. These cargo cults are completely different from the description of cargo cults you usually find on the internet, which I'll call the "pop-culture cargo cult." Cargo cults are extremely diverse, to the extent that anthropologists disagree on the cause, definition, or even if the term has value. I'll show that many of the popular views of cargo cults come from a 1962 "shockumentary" called Mondo Cane. Moreover, most online photos of cargo cults are fake.
The cargo cult metaphor in science started with Professor Richard Feynman's well-known 1974 commencement address at Caltech.1 This speech, titled "Cargo Cult Science", was expanded into a chapter in his best-selling 1985 book "Surely You're Joking, Mr. Feynman". He said:
In the South Seas there is a cargo cult of people. During the war they saw airplanes land with lots of good materials, and they want the same thing to happen now. So they’ve arranged to make things like runways, to put fires along the sides of the runways, to make a wooden hut for a man to sit in, with two wooden pieces on his head like headphones and bars of bamboo sticking out like antennas—he’s the controller—and they wait for the airplanes to land. They’re doing everything right. The form is perfect. It looks exactly the way it looked before. But it doesn’t work. No airplanes land. So I call these things cargo cult science, because they follow all the apparent precepts and forms of scientific investigation, but they’re missing something essential, because the planes don’t land.
But the standard anthropological definition of "cargo cult" is entirely different: 2
Cargo cults are strange religious movements in the South Pacific that appeared during the last few decades. In these movements, a prophet announces the imminence of the end of the world in a cataclysm which will destroy everything. Then the ancestors will return, or God, or some other liberating power, will appear, bringing all the goods the people desire, and ushering in a reign of eternal bliss.
An anthropology encyclopedia gives a similar definition:
A southwest Pacific example of messianic or millenarian movements once common throughout the colonial world, the modal cargo cult was an agitation or organised social movement of Melanesian villagers in pursuit of ‘cargo’ by means of renewed or invented ritual action that they hoped would induce ancestral spirits or other powerful beings to provide. Typically, an inspired prophet with messages from those spirits persuaded a community that social harmony and engagement in improvised ritual (dancing, marching, flag-raising) or revived cultural traditions would, for believers, bring them cargo.
As you may see, the pop-culture explanation of a cargo cult and the anthropological definition are completely different, apart from the presence of "cargo" of some sort. Have anthropologists buried cargo cults under layers of theory? Are they even discussing the same thing? My conclusion, after researching many primary sources, is that the anthropological description accurately describes the wide variety of cargo cults. The pop-culture cargo cult description, however, takes features of some cargo cults (the occasional runway) and combines this with movie scenes to yield an inaccurate and fictionalized dscription. It may be hard to believe that the description of cargo cults that you see on the internet is mostly wrong, but in the remainder of this article, I will explain this in detail.
Cargo cults occur in a specific region of the South Pacific called Melanesia. I'll give a brief (oversimplified) description of Melanesia to provide important background. The Pacific Ocean islands are divided into three cultural areas: Polynesia, Micronesia, and Melanesia. Polynesia is the best known, including Hawaii, New Zealand, and Samoa. Micronesia, in the northwest, consists of thousands of small islands, of which Guam is the largest; the name "Micronesia" is Greek for "small island". Melanesia, the relevant area for this article, is a group of islands between Micronesia and Australia, including Fiji, Vanuatu, Solomon Islands, and New Guinea. (New Guinea is the world's second-largest island; confusingly, the country of Papua New Guinea occupies the eastern half of the island, while the western half is part of Indonesia.)
The inhabitants of Melanesia typically lived in small villages of under 200 people, isolated by mountainous geography. They had a simple, subsistence economy, living off cultivated root vegetables, pigs, and hunting. People tended their own garden, without specialization into particular tasks. The people of Melanesia are dark-skinned, which will be important ("Melanesia" and "melanin" have the same root). Technologically, the Melanesians used stone, wood, and shell tools, without knowledge of metallurgy or even weaving. The Melanesian cultures were generally violent3 with everpresent tribal warfare and cannibalism.4
Due to the geographic separation of tribes, New Guinea became the most linguistically diverse country in the world, with over 800 distinct languages. Pidgin English was often the only way for tribes to communicate, and is now one of the official languages of New Guinea. This language, called Tok Pisin (i.e. "talk pidgin"), is now the most common language in Papua New Guinea, spoken by over two-thirds of the population.5
For the Melanesians, religion was a matter of ritual, rather than a moral framework. It is said that "to the Melanesian, a religion is above all a technology: it is the knowledge of how to bring the community into the correct relation, by rites and spells, with the divinities and spirit-beings and cosmic forces that can make or mar man's this-worldly wealth and well-being." This is important since, as will be seen, the Melanesians expected that the correct ritual would result in the arrival of cargo. Catholic and Protestant missionaries converted the inhabitants to Christianity, largely wiping out traditional religious practices and customs; Melanesia is now over 95% Christian. Christianity played a large role in cargo cults, as will be shown below.
European explorers first reached Melanesia in the 1500s, followed by colonization.6 By the end of the 1800s, control of the island of New Guinea was divided among Germany, Britain, and the Netherlands. Britain passed responsibility to Australia in 1906 and Australia gained the German part of New Guinea in World War I. As for the islands of Vanuatu, the British and French colonized them (under the name New Hebrides) in the 18th century.
The influx of Europeans was highly harmful to the Melanesians. "Native society was severely disrupted by war, by catastrophic epidemics of European diseases, by the introduction of alcohol, by the devastation of generations of warfare, and by the depredations of the labour recruiters."8 People were kidnapped and forced to work as laborers in other countries, a practice called blackbirding. Prime agricultural land was taken by planters to raise crops such as coconuts for export, with natives coerced into working for the planters.9 Up until 1919, employers were free to flog the natives for disobedience; afterward, flogging was technically forbidden but still took place. Colonial administrators jailed natives who stepped out of line.7
While the pop-culture cargo cults explains them as a reaction to World War II, cargo cults started years earlier. One anthropologist stated, "Cargo cults long preceded [World War II], continued to occur during the war, and have continued to the present."
The first writings about cargo cult behavior date back to 1919, when it was called the "Vailala Madness":10
The natives were saying that the spirits of their ancestors had appeared to several in the villages and told them that all flour, rice, tobacco, and other trade belonged to the New Guinea people, and that the white man had no right whatever to these goods; in a short time all the white men were to be driven away, and then everything would be in the hands of the natives; a large ship was also shortly to appear bringing back the spirits of their departed relatives with quantities of cargo, and all the villages were to make ready to receive them.
The 1926 book In Unknown New Guinea also describes the Vialala Madness:11
[The leader proclaimed] that the ancestors were coming back in the persons of the white people in the country and that all the things introduced by the white people and the ships that brought them belonged really to their ancestors and themselves. [He claimed that] he himself was King George and his friend was the Governor. Christ had given him this authority and he was in communication with Christ through a hole near his village.
The Melanesians blamed the Europeans for the failure of cargo to arrive. In the 1930s, one story was that because the natives had converted to Christianity, God was sending the ancestors with cargo that was loaded on ships. However, the Europeans were going through the cargo holds and replacing the names on the crates so the cargo was fraudulently delivered to the Europeans instead of the rightful natives.
The Mambu Movement occurred in 1937. Mambu, the movement's prophet, claimed that "the Whites had deceived the natives. The ancestors lived inside a volcano on Manum Island, where they worked hard making goods for their descendants: loin-cloths, socks, metal axes, bush-knives, flashlights, mirrors, red dye, etc., even plank-houses, but the scoundrelly Whites took the cargoes. Now this was to stop. The ancestors themselves would bring the goods in a large ship." To stop this movement, the Government arrested Mambu, exiled him, and imprisoned him for six months in 1938.
To summarize, these early cargo cults believed that ships would bring cargo that rightfully belonged to the natives but had been stolen by the whites. The return of the cargo would be accompanied by the spirits of the ancestors. Moreover, Christianity often played a large role. A significant racial component was present, with natives driving out the whites or becoming white themselves.
World War II caused tremendous social and economic upheavals in Melanesia. Much of Melanesia was occupied by Japan near the beginning of the war and the Japanese treated the inhabitants harshly. The American entry into the war led to heavy conflict in the area such as the arduous New Guinea campaign (1942-1945) and the Solomon Islands campaign. As the Americans and Japanese battled for control of the islands, the inhabitants were caught in the middle. Papua and New Guinea suffered over 15,000 civilian deaths, a shockingly high number for such a small region.12
The impact of the Japanese occupation on cargo cults is usually ignored. One example from 1942 is a cargo belief that the Japanese soldiers were spirits of the dead, who were being sent by Jesus to liberate the people from European rule. The Japanese would bring the cargo by airplane since the Europeans were blocking the delivery of cargo by ship. This would be accompanied by storms and earthquakes, and the natives' skin would change from black to white. The natives were to build storehouses for the cargo and fill the storehouses with food for the ancestors. The leader of this movement, named Tagarab, explained that he had an iron rod that gave him messages about the future. Eventually, the Japanese shot Tagarab, bringing an end to this cargo cult.13
The largest and most enduring cargo cult is the John Frum movement, which started on the island of Tanna around 1941 and continues to the present. According to one story, a mythical person known as John Frum, master of the airplanes, would reveal himself and drive off the whites. He would provide houses, clothes, and food for the people of Tanna. The island of Tanna would flatten as the mountains filled up the valleys and everyone would have perfect health. In other areas, the followers of John Frum believed they "would receive a great quantity of goods, brought by a white steamer which would come from America." Families abandoned the Christian villages and moved to primitive shelters in the interior. They wildly spent much of their money and threw the rest into the sea. The government arrested and deported the leaders, but that failed to stop the movement. The identity of John Frum is unclear; he is sometimes said to be a white American while in other cases natives have claimed to be John Frum.14
The cargo cult of Kainantu17 arose around 1945 when a "spirit wind" caused people in the area to shiver and shake. Villages built large "cargo houses" and put stones, wood, and insect-marked leaves inside, representing European goods, rifles, and paper letters respectively. They killed pigs and anointed the objects, the house, and themselves with blood. The cargo house was to receive the visiting European spirit of the dead who would fill the house with goods. This cargo cult continued for about 5 years, diminishing as people became disillusioned by the failure of the goods to arrive.
The name "Cargo Cult" was first used in print in 1945, just after the end of World War II.15 The article blamed the problems on the teachings of missionaries, with the problems "accentuated a hundredfold" by World War II.
Stemming directly from religious teaching of equality, and its resulting sense of injustice, is what is generally known as “Vailala Madness,” or “Cargo Cult.” "In all cases the "Madness" takes the same form: A native, infected with the disorder, states that he has been visited by a relative long dead, who stated that a great number of ships loaded with "cargo" had been sent by the ancestor of the native for the benefit of the natives of a particular village or area. But the white man, being very cunning, knows how to intercept these ships and takes the "cargo" for his own use... Livestock has been destroyed, and gardens neglected in the expectation of the magic cargo arriving. The natives infected by the "Madness" sank into indolence and apathy regarding common hygiene."
In a 1946 episode, agents of the Australian government found a group of New Guinea highlanders who believed that the arrival of the whites signaled that the end of the world was at hand. The highlanders butchered all their pigs in the expectation that "Great Pigs" would appear from the sky in three days. At this time, the residents would exchange their black skin for white skin. They created mock radio antennas of bamboo and rope to receive news of the millennium.16
The New York Times described Cargo Cults in 1948 as "the belief that a convoy of cargo ships is on its way, laden with the fruits of the modern world, to outfit the leaf huts of the natives." The occupants of the British Solomon Islands were building warehouses along the beaches to hold these goods. Natives marched into a US Army camp, presented $3000 in US money, and asked the Army to drive out the British.
A 1951 paper described cargo cults: "The insistence that a 'cargo' of European goods is to be sent by the ancestors or deceased spirits; this may or may not be part of a general reaction against Europeans, with an overtly expressed desire to be free from alien domination. Usually the underlying theme is a belief that all trade goods were sent by ancestors or spirits as gifts for their descendants, but have been misappropriated on the way by Europeans."17
In 1959, The New York Times wrote about cargo cults: "Rare Disease and Strange Cult Disturb New Guinea Territory; Fatal Laughing Sickness Is Under Study by Medical Experts—Prophets Stir Delusions of Food Arrivals". The article states that "large native groups had been infected with the idea that they could expect the arrival of spirit ships carrying large supplies of food. In false anticipation of the arrival of the 'cargoes', 5000 to 7000 native have been known to consume their entire food reserve and create a famine." As for "laughing sickness", this is now known to be a prion disease transmitted by eating human brains. In some communities, this disease, also called Kuru, caused 50% of all deaths.
A detailed 1959 article in Scientific American, "Cargo Cults", described many different cargo cults.16 It lists various features of cargo cults, such as the return of the dead, skin color switching from black to white, threats against white rule, and belief in a coming messiah. The article finds a central theme in cargo cults: "The world is about to end in a terrible cataclysm. Thereafter God, the ancestors or some local culture hero will appear and inaugurate a blissful paradise on earth. Death, old age, illness and evil will be unknown. The riches of the white man will accrue to the Melanesians."
In 1960, the celebrated naturalist David Attenborough created a documentary The People of Paradise: Cargo Cult.18 Attenborough travels through the island of Tanna and encounters many artifacts of the John Frum cult, such as symbolic gates and crosses, painted brilliant scarlet and decorated with objects such as a shaving brush, a winged rat, and a small carved airplane. Attenborough interviews a cult leader who claims to have talked with the mythical John Frum, said to be a white American. The leader remains in communication with John Frum through a tall pole said to be a radio mast, and an unseen radio. (The "radio" consisted of an old woman with electrical wire wrapper around her waist, who would speak gibberish in a trance.)
In 1963, famed anthropologist Margaret Mead brought cargo cults to the general public, writing Where Americans are Gods: The Strange Story of the Cargo Cults in the mass-market newspaper supplement Family Weekly. In just over a page, this article describes the history of cargo cults before, during, and after World War II.19 One cult sat around a table with vases of colorful flowers on them. Another cult threw away their money. Another cult watched for ships from hilltops, expecting John Frum to bring a fleet of ships bearing cargo from the land of the dead.
One of the strangest cargo cults was a group of 2000 people on New Hanover Island, "collecting money to buy President Johnson of the United States [who] would arrive with other Americans on the liner Queen Mary and helicopters next Tuesday." The islanders raised $2000, expecting American cargo to follow the president. Seeing the name Johnson on outboard motors confirmed their belief that President Johnson was personally sending cargo.20
A 1971 article in Time Magazine22 described how tribesmen brought US Army concrete survey markers down from a mountaintop while reciting the Roman Catholic rosary, dropping the heavy markers outside the Australian government office. They expected that "a fleet of 500 jet transports would disgorge thousands of sympathetic Americans bearing crates of knives, steel axes, rifles, mirrors and other wonders." Time magazine explained the “cargo cult” as "a conviction that if only the dark-skinned people can hit on the magic formula, they can, without working, acquire all the wealth and possessions that seem concentrated in the white world... They believe that everything has a deity who has to be contacted through ritual and who only then will deliver the cargo." Cult leaders tried "to duplicate the white man’s magic. They hacked airstrips in the rain forest, but no planes came. They built structures that look like white men’s banks, but no money materialized."21
National Geographic, in an article Head-hunters in Today's World (1972), mentioned a cargo-cult landing field with a replica of a radio aerial, created by villagers who hoped that it would attract airplanes bearing gifts. It also described a cult leader in South Papua who claimed to obtain airplanes and cans of food from a hole in the ground. If the people believed in him, their skins would turn white and he would lead them to freedom.
These sources and many others23 illustrate that cargo cults do not fit a simple story. Instead, cargo cults are extremely varied, happening across thousands of miles and many decades. The lack of common features between cargo cults leads some anthropologists to reject the idea of cargo cults as a meaningful term.24 In any case, most historical cargo cults have very little in common with the pop-culture description of a cargo cult.
Cargo cult beliefs are closely tied to Christianity, a factor that is ignored in pop-culture descriptions of cargo cults. Beginning in the mid-1800s, Christian missionaries set up churches in New Guinea to convert the inhabitants. As a result, cargo cults incorporated Christian ideas, but in very confusing ways. At first, the natives believed that missionaries had come to reveal the ritual secrets and restore the cargo. By enthusiastically joining the church, singing the hymns, and following the church's rituals, the people would be blessed by God, who would give them the cargo. This belief was common in the 1920s and 1930s, but as the years went on and the people didn't receive the cargo, they theorized that the missionaries had removed the first pages of the Bible to hide the cargo secrets.
A typical belief was that God created Adam and Eve in Paradise, "giving them cargo: tinned meat, steel tools, rice in bags, tobacco in tins, and matches, but not cotton clothing." When Adam and Eve offended God by having sexual intercourse, God threw them out of Paradise and took their cargo. Eventually, God sent the Flood but Noah was saved in a steamship and God gave back the cargo. Noah's son Ham offended God, so God took the cargo away from Ham and sent him to New Guinea, where he became the ancestor of the natives.
Other natives believed that God lived in Heaven, which was in the clouds and reachable by ladder from Sydney, Australia (source). God, along with the ancestors, created cargo in Heaven—"tinned meat, bags of rice, steel tools, cotton cloth, tinned tobacco, and a machine for making electric light"—which would be flown from Sydney and delivered to the natives, who thus needed to clear an airstrip (source).25
Another common belief was that symbolic radios could be used to communicate with Jesus. For instance, a Markham Valley cargo group in 1943 created large radio houses so they could be informed of the imminent Coming of Jesus, at which point the natives would expel the whites (source). The "radio" consisted of bamboo cylinders connected to a rope "aerial" strung between two poles. The houses contained a pole with rungs so the natives could climb to Jesus along with cane "flashlights" to see Jesus.
Mock radio antennas are also discussed in a 1943 report26 from a wartime patrol that found a bamboo "wireless house", 42 feet in diameter. It had two long poles outside and with an "aerial" of rope between them, connected to the "radio" inside, a bamboo cylinder. Villagers explained that the "radio" was to receive messages of the return of Jesus, who would provide weapons for the overthrow of white rule. The villagers constructed ladders outside the house so they could climb up to the Christian God after death. They would shed their skin like a snake, getting a new white skin, and then they would receive the "boats and white men's clothing, goods, etc."
As described above, cargo cults expected the cargo to arrive by ships much more often than airplanes. So why do pop-culture cargo cults have detailed descriptions of runways, airplanes, wooden headphones, and bamboo control towers?27 My hypothesis is that it came from a 1962 movie called Mondo Cane. This film was the first "shockumentary", showing extreme and shocking scenes from around the world. Although the film was highly controversial, it was shown at the Cannes Film Festival and was a box-office success.
The film made extensive use of New Guinea with multiple scandalous segments, such as a group of "love-struck" topless women chasing men,29 a woman breastfeeding a pig, and women in cages being fattened for marriage. The last segment in the movie showed "the cult of the cargo plane": natives forlornly watching planes at the airport, followed by scenes of a bamboo airplane sitting on a mountaintop "runway" along with bamboo control towers. The natives waited all day and then lit torches to illuminate the runway at nightfall. These scenes are very similar to the pop-culture descriptions of cargo cults so I suspect this movie is the source.
The film claims that all the scenes "are true and taken only from life", but many of the scenes are said to be staged. Since the cargo cult scenes are very different from anthropological reports and much more dramatic, I think they were also staged and exaggerated.28 It is known that the makers of Mondo Cane paid the Melanesian natives generously for the filming (source, source).
Did Feynman get his cargo cult ideas from Mondo Cane? It may seem implausible since the movie was released over a decade earlier. However, the movie became a cult classic, was periodically shown in theaters, and influenced academics.30 In particular, Mondo Cane showed at the famed Cameo theater in downtown Los Angeles on April 3, 1974, two months before Feynman's commencement speech. Mondo Cane seems like the type of offbeat movie that Feynman would see and the theater was just 11 miles from Caltech. While I can't prove that Feynman went to the showing, his description of a cargo cult strongly resembles the movie.31
Fakes and hoaxes make researching cargo cults online difficult. There are numerous photos online of cargo cults, but many of these photos are completely made up. For instance, the photo below has illustrated cargo cults for articles such as Cargo Cult, UX personas are useless, A word on cargo cults, The UK Integrated Review and security sector innovation, and Don't be a cargo cult. However, this photo is from a Japanese straw festival and has nothing to do with cargo cults.
Another example is the photo below, supposedly an antenna created by a cargo cult. However, it is actually a replica of the Jodrell Bank radio telescope, built in 2007 by a British farmer from six tons of straw (details). The farmer's replica ended up erroneously illustrating Cargo Cult Politics, The Cargo Cult & Beliefs, The Cargo Cult, Cargo Cults of the South Pacific, and Cargo Cult, among others.32
Other articles illustrate cargo cults with the aircraft below, suspiciously sleek and well-constructed. However, the photo actually shows a wooden wind tunnel model of the Buran spacecraft, abandoned at a Russian airfield as described in this article. Some uses of the photo are Are you guilty of “cargo cult” thinking without even knowing it? and The Cargo Cult of Wealth.
Many cargo cult articles use one of the photo below. I tracked them down to the 1970 movie "Chariots of the Gods" (link), a dubious documentary claiming that aliens have visited Earth throughout history. The segment on cargo cults is similar to Mondo Cane with cultists surrounding a mock plane on a mountaintop, lighting fires along the runway. However, it is clearly faked, probably in Africa: the people don't look like Pacific Islanders and are wearing wigs. One participant wears leopard skin (leopards don't live in the South Pacific). The vegetation is another giveaway: the plants are from Africa, not the South Pacific.33
The point is that most of the images that illustrate cargo cults online are fake or wrong. Most internet photos and information about cargo cults have just been copied from page to page. (And now we have AI-generated cargo cult photos.) If a photo doesn't have a clear source (including who, when, and where), don't believe it.
The cargo cult metaphor should be avoided for three reasons. First, the metaphor is essentially meaningless and heavily overused. The influential "Jargon File" defined cargo-cult programming as "A style of (incompetent) programming dominated by ritual inclusion of code or program structures that serve no real purpose."34 Note that the metaphor in cargo-cult programming is the opposite of the metaphor in cargo-cult science: Feyman's cargo-cult science has no chance of working, while cargo-cult programming works but isn't understood. Moreover, both metaphors differ from the cargo-cult metaphor in other contexts, referring to the expectation of receiving valuables without working.35
The popular site Hacker News is an example of how "cargo cult" can be applied to anything: agile programming, artificial intelligence, cleaning your desk. Go, hatred of Perl, key rotation, layoffs, MBA programs, microservices, new drugs, quantum computing, static linking, test-driven development, and updating the copyright year are just a few things that are called "cargo cult".36 At this point, cargo cult is simply a lazy, meaningless attack.
The second problem with "cargo cult" is that the pop-culture description of cargo cults is historically inaccurate. Actual cargo cults are much more complex and include a much wider (and stranger) variety of behaviors. Cargo cults started before World War II and involve ships more often than airplanes. Cargo cults mix aspects of paganism and Christianity, often with apocalyptic ideas of the end of the current era, the overthrow of white rule, and the return of dead ancestors. The pop-culture description discards all this complexity, replacing it with a myth.
Finally, the cargo cult metaphor turns decades of harmful colonialism into a humorous anecdote. Feynman's description of cargo cults strips out the moral complexity: US soldiers show up with their cargo and planes, the indigenous residents amusingly misunderstand the situation, and everyone carries on. However, cargo cults really were a response to decades of colonial mistreatment, exploitation, and cultural destruction. Moreover, cargo cults were often harmful: expecting a bounty of cargo, villagers would throw away their money, kill their pigs, and stop tending their crops, resulting in famine. The pop-culture cargo cult erases the decades of colonial oppression, along with the cultural upheaval and deaths from World War II. Melanesians deserve to be more than the punch line in a cargo cult story.
Thus, it's time to move beyond the cargo cult metaphor.
Update: well, this sparked much more discussion on Hacker News than I expected. To answer some questions: Am I better or more virtuous than other people? No. Are you a bad person if you use the cargo cult metaphor? No. Is "cargo cult" one of many Hacker News comments that I'm tired of seeing? Yes (details). Am I criticizing Feynman? No. Do the Melanesians care about this? Probably not. Did I put way too much research into this? Yes. Is criticizing colonialism in the early 20th century woke? I have no response to that.
As an illustration of the popularity of Feynman's "Cargo Cult Science" commencement address, it has been on Hacker News at least 15 times. ↩
The first cargo cult definition above comes from The Trumpet Shall Sound; A Study of "Cargo" Cults in Melanesia. The second definition is from the Cargo Cult entry in The Open Encyclopedia of Anthropology. Written by Lamont Lindstrom, a professor who studies Melanesia, the entry comprehensively describes the history and variety of cargo cults, as well as current anthropological analysis.
For an early anthropological theory of cargo cults, see An Empirical Case-Study: The Problem of Cargo Cults in "The Revolution in Anthropology" (Jarvie, 1964). This book categorizes cargo cults as an apocalyptic millenarian religious movement with a central tenet:
When the millennium comes it will largely consist of the arrival of ships and/or aeroplanes loaded up with cargo; a cargo consisting either of material goods the natives long for (and which are delivered to the whites in this manner), or of the ancestors, or of both.↩
European colonization brought pacification and a reduction in violence. The Cargo Cult: A Melanesian Type-Response to Change describes this pacification and termination of warfare as the Pax Imperii, suggesting that pacification came as a relief to the Melanesians: "They welcomed the cessation of many of the concomitants of warfare: the sneak attack, ambush, raiding, kidnapping of women and children, cannibalism, torture, extreme indignities inflicted on captives, and the continual need to be concerned with defense." That article calls the peace the Pax Imperii.
Warfare among the Enga people of New Guinea is described in From Spears to M-16s: Testing the Imbalance of Power Hypothesis among the Enga. The Enga engaged in tribal warfare for reasons such as "theft of game from traps, quarrels over possessions, or work sharing within the group." The surviving losers were usually driven off the land and forced to settle elsewhere. In the 1930s and 1940s, the Australian administration banned tribal fighting and pacified much of the area. However, after the independence of Papua New Guinea in 1975, warfare increased along with the creation of criminal gangs known as Raskols (rascals). The situation worsened in the late 1980s with the introduction of shotguns and high-powered weapons to warfare. Now, Papua New Guinea has one of the highest crime rates in the world along with one of the lowest police-to-population ratios in the world. ↩
When you hear tales of cannibalism, some skepticism is warranted. However, cannibalism is proved by the prevalence of kuru, or "laughing sickness", a fatal prion disease (transmissible spongiform encephalopathy) spread by consuming human brains. Also see Headhunters in Today's World, a 1972 National Geographic article that describes the baking of heads and the eating of brains. ↩
A 1957 dictionary of Pidgin English can be found here. Linguistically, Tok Pisin is a creole, not a pidgin. ↩
The modern view is that countries such as Great Britain acquired colonies against the will of the colonized, but the situation was more complex in the 19th century. Many Pacific islands desperately wanted to become European colonies, but were turned down for years because the countries were viewed as undesiable burdens.
For example, Fiji viewed colonization as the solution to the chaos caused by the influx of white settlers in the 1800s. Fijian political leaders attempted to cede the islands to a European power that could end the lawlessness, but were turned down. In 1874, the situation changed when Disraeli was elected British prime minister. His pro-imperial policies, along with the Royal Navy's interest in obtaining a coaling station, concerns about American expansion, and pressure from anti-slavery groups, led to the annexation of Fiji by Britain. The situation in Fiji didn't particularly improve from annexation. (Fiji obtained independence almost a century later, in 1970.)
As an example of the cost of a colony, Australia was subsidizing Papua New Guinea (with a population of 2.5 million) with over 100 million dollars a year in the early 1970s. (source) ↩
When reading about colonial Melanesia, one notices a constant background of police activity. Even when police patrols were very rare (annual in some parts), they were typically accompanied by arbitrary arrests and imprisonment. The most common cause for arrest was adultery; it may seem strange that the police were so concerned with it, but it turns out that adultery was the most common cause of warfare between tribes, and the authorities were trying to reduce the level of warfare. Cargo cult activity could be punished by six months of imprisonment. Jailing tended to be ineffective in stopping cargo cults, however, as it was viewed as evidence that the Europeans were trying to stop the cult leaders from spreading the cargo secrets that they had uncovered. ↩
See The Trumpet Shall Sound. ↩
The government imposed a head tax, which for the most part could only be paid through employment. A 1924 report states, "The primary object of the head tax was not to collect revenue but to create among the natives a need for money, which would make labour for Europeans desirable and would force the natives to accept employment." ↩
The Papua Annual Report, 1919-20 includes a report on the "Vailala Madness", starting on page 118. It describes how villages with the "Vialala madness" had "ornamented flag-poles, long tables, and forms or benches, the tables being usually decorated with flowers in bottles of water in imitation of a white man's dining table." Village men would sit motionless with their backs to the tables. Their idleness infuriated the white men, who considered the villagers to be "fit subjects for a lunatic asylum." ↩
The Vailala Madness is also described in The Missionary Review of the World, 1924. The Vaialala Madness also involved seizure-like physical aspects, which typically didn't appear in later cargo cult behavior.
The 1957 book The Trumpet Shall Sound: A Study of "Cargo" Cults in Melanesia is an extensive discussion of cargo cults, as well as earlier activity and movements. Chapter 4 covers the Vailala Madness in detail. ↩
The battles in the Pacific have been extensively described from the American and Japanese perspectives, but the indigenous residents of these islands are usually left out of the narratives. This review discusses two books that provide the Melanesian perspective.
I came across the incredible story of Sergeant Major Vouza of the Native Constabulary. While this story is not directly related to cargo cults, I wanted to include it as it illustrates the dedication and suffering of the New Guinea natives during World War II. Vouza volunteered to scout behind enemy lines for the Marines at Guadalcanal but he was captured by the Japanese, tied to a tree, tortured, bayonetted, and left for dead. He chewed through his ropes, made his way through the enemy force, and warned the Marines of an impending enemy attack.
Vouza described the event in a letter:
The Japanese occupation and the cargo cult started by Tagareb are described in detail in Road Belong Cargo, pages 98-110. (An entertaining review of that book is here.) ↩
See "John Frum Movement in Tanna", Oceania, March 1952. The New York Times described the John Frum movement in detail in a 1970 article: "On a Pacific island, they wait for the G.I. who became a God". A more modern article (2006) on John Frum is In John They Trust in the Smithsonian Magazine.
As for the identity of John Frum, some claim that his name is short for "John from America". Others claim it is a modification of "John Broom" who would sweep away the whites. These claims lack evidence. ↩
The quote is from Pacific Islands Monthly, November 1945 (link). The National Library of Australia has an extensive collection of issues of Pacific Islands Monthly online. Searching these magazines for "cargo cult" provides an interesting look at how cargo cults were viewed as they happened. ↩
Scientific American had a long article titled Cargo Cults in May 1959, written by Peter Worsley, who also wrote the classic book The Trumpet Shall Sound: A Study of 'Cargo' Cults in Melanesia. The article lists the following features of cargo cults:
Different cargo cults contained different subsets of these features but no specific feature The article is reprinted here; the detailed maps show the wide distribution of cargo cults. ↩↩
See A Cargo Movement in the Eastern Central Highlands of New Guinea, Oceania, 1952. ↩↩
The Attenborough Cargo Cult documentary can be watched on YouTube.
I'll summarize some highlights with timestamps:
5:20: A gate, palisade, and a cross all painted brilliant red.
6:38: A cross decorated with a wooden bird and a shaving brush.
7:00: A tall pole claimed to be a special radio mast to talk with John Frum.
8:25: Interview with trader Bob Paul. He describes "troops" marching with wooden guns around the whole island.
12:00: Preparation and consumption of kava, the intoxicating beverage.
13:08: Interview with a local about John Frum.
14:16: John Frum described as a white man and a big fellow.
16:29: Attenborough asks, "You say John Frum has not come for 19 years. Isn't this a long time for you to wait?"
The leader responds, "No, I can wait. It's you waiting for two thousand years for Christ to come and I must wait over 19 years."
Attenborough accepts this as a fair point.
17:23: Another scarlet gate, on the way to the volcano, with a cross, figure, and model airplane.
22:30: Interview with the leader. There's a discussion of the radio, but Attenborough is not allowed to see it.
24:21: John Frum is described as a white American.
The expedition is also described in David Attenborough's 1962 book Quest in Paradise. ↩
I have to criticize Mead's article for centering Americans as the heroes, almost a parody of American triumphalism. The title sets the article's tone: "Where Americans are Gods..." The article explains, "The Americans were lavish. They gave away Uncle Sam's property with a generosity which appealed mightily... so many kind, generous people, all alike, with such magnificent cargoes! The American servicemen, in turn, enjoyed and indulged the islanders."
The article views cargo cults as a temporary stage before moving to a prosperous American-style society as islanders realized that "American things could come [...] only by work, education, persistence." A movement leader named Paliau is approvingly quoted: "We would like to have the things Americans have. [...] We think Americans have all these things because they live under law, without endless quarrels. So we must first set up a new society."
On the other hand, by most reports, the Americans treated the residents of Melanesia much better than the colonial administrators. Americans paid the natives much more (which was viewed as overpaying them by the planters). The Americans treated the natives with much more respect; natives worked with Americans almost as equals. Finally, it appeared to the natives that black soldiers were treated as equals to white soldiers. (Obviously, this wasn't entirely accurate.)
The Melanesian experience with Americans also strengthened Melanesian demands for independence. Following the war, the reversion to colonial administration produced a lot of discontent in the natives, who realized that their situation could be much better. (See World War II and Melanesian self-determination.) ↩
The Johnson cult was analyzed in depth by Billings, an anthropologist who wrote about it in Cargo Cult as Theater: Political Performance in the Pacific. See also Australian Daily News, June 12, 1964, and Time Magazine, July 19, 1971. ↩
In one unusual case, the islanders built an airstrip and airplanes did come. Specifically, the Miyanmin people of New Guinea hacked an airstrip out of the forest in 1966 using hand tools. The airstrip was discovered by a patrol and turned out to be usable, so Baptist missionaries made monthly landings, bringing medicine and goods for a store. It is pointed out that the only thing preventing this activity from being considered a cargo cult is that in this case, it was effective. See A Small Footnote to the 'Big Walk', p. 59. ↩
See "New Guinea: Waiting for That Cargo", Time Magazine, July 19, 1971. ↩
In this footnote, I'll list some interesting cargo cult stories that didn't fit into the body of the article.
The 1964 US Bureau of Labor Statistics report on New Guinea describes cargo cults: "A simplified explanation of them is often given namely that contact with Western culture has given the indigene a desire for a better economic standard of living this desire has not been accompanied by the understanding that economic prosperity is achieved by human effort. The term cargo cult derives from the mystical expectation of the imminent arrival by sea or air of the good things of this earth. It is believed sufficient to build warehouses of leaves and prepare air strips to receive these goods. Activity in the food gardens and daily community routine chores is often neglected so that economic distress is engendered."
Cargo Cult Activity in Tangu (Burridge) is a 1954 anthropological paper discussing stories of three cargo cults in Tangu, a region of New Guinea. The first involved dancing around a man in a trance, which was supposed to result in the appearance of "rice, canned meat, lava-lavas, knives, beads, etc." In the second story, villagers built a shed in a cemetery and then engaged in ritualized sex acts, expecting the shed to be filled with goods. However, the authorities forced the participants to dismantle the shed and throw it into the sea. In the third story, the protagonist is Mambu, who stowed away on a steamship to Australia, where he discovered the secrets of the white man's cargo. On his return, he collected money to help force the Europeans out, until he was jailed. He performed "miracles" by appearing outside jail as well as by producing money out of thin air.
Reaction to Contact in the Eastern Highlands of New Guinea (Berndt, 1954) has a long story about Berebi, a leader who was promised a rifle, axes, cloth, knives, and valuable cowrie by a white spirit. Berebi convinces his villagers to build storehouses and they filled the houses with stones that would be replaced by goods. They take part in many pig sacrifices and various rituals, and endure attacks of shivering and paralysis, but they fail to receive any goods and Berebi concludes that the spirit deceived him. ↩
Many anthropologists view the idea of cargo cults as controversial. One anthropologist states, "What I want to suggest here is that, similarly, cargo cults do not exist, or at least their symptoms vanish when we start to doubt that we can arbitrarily extract a few features from context and label them an institution." See A Note on Cargo Cults and Cultural Constructions of Change (1988). The 1992 paper The Yali Movement in Retrospect: Rewriting History, Redefining 'Cargo Cult' summarizes the uneasiness that many anthropologists have with the term "cargo cult", viewing it as "tantamount to an invocation of colonial power relationships."
The book Cargo, Cult, and Culture Critique (2004) states, "Some authors plead quite convincingly for the abolition of the term itself, not only because of its troublesome implications, but also because, in their view, cargo cults do not even exist as an identifiable object of study." One paper states that the phrase is both inaccurate and necessary, proposing that it be written crossed-out (sous rature in Derrida's post-modern language). Another paper states: "Cargo cults defy definition. They are inherently troublesome and problematic," but concludes that the term is useful precisely because of this troublesome nature.
At first, I considered the idea of abandoning the label "cargo cult" to be absurd, but after reading the anthropological arguments, it makes more sense. In particular, the category "cargo cult" is excessively broad, lumping together unrelated things and forcing them into a Procrustean ideal: John Frum has very little in common with Vaialala Madness, let alone the Johnson Cult. I think that the term "cargo cult" became popular due to its catchy, alliterative name. (Journalists love alliterations such as "Digital Divide" or "Quiet Quitting".) ↩
It was clear to the natives that the ancestors, and not the Europeans, must have created the cargo because the local Europeans were unable to repair complex mechanical devices locally, but had to ship them off. These ships presumably took the broken devices back to the ancestral spirits to be repaired. Source: The Trumpet Shall Sound, p119. ↩
The report from the 1943 patrol is discussed in Berndt's "A Cargo Movement in the Eastern Central Highlands of New Guinea", Oceania, Mar. 1953 (link), page 227. These radio houses are also discussed in The Trumpet Shall Sound, page 199. ↩
Wooden airplanes are a staple of the pop-culture cargo cult story, but they are extremely rare in authentic cargo cults. I searched extensively, but could find just a few primary sources that involve airplanes.
The closest match that I could find is Vanishing Peoples of the Earth, published by National Geographic in 1968, which mentions a New Guinea village that built a "crude wooden airplane", which they thought "offers the key to getting cargo".
The photo below, from 1950, shows a cargo-house built in the shape of an airplane. (Note how abstract the construction is, compared to the realistic straw airplanes in faked photos.) The photographer mentioned that another cargo house was in the shape of a jeep, while in another village, the villagers gather in a circle at midnight to await the arrival of heavily laden cargo boats.
David Attenborough's Cargo Cult documentary shows a small wooden airplane, painted scarlet red. This model airplane is very small compared to the mock airplanes described in the pop-culture cargo cult.
The photo below shows the scale of the aircraft, directly in front of Attenborough. In the center, a figure of John Frum has a "scarlet coat and a white, European face." On the left, a cage contains a winged rat for some reason.
The photo below shows another scene from the movie Mondo Cane that is very popular online in cargo cult articles. I suspect that the airplane is not authentic but was made for the movie.
The tale of women pursuing men was described in detail in the 1929 anthropological book The Sexual Life of Savages in North-Western Melanesia, specifically the section "Yausa—Orgiastic Assaults by Women" (pages 231-234). The anthropologist heard stories about these attacks from natives, but didn't observe them firsthand and remained skeptical. He concluded that "The most that can be said with certainty is that the yausa, if it happened at all, happened extremely rarely". Unlike the portrayal in Mondo Cane, these attacks on men were violent and extremely unpleasant (I won't go into details). Thus, it is very likely that this scene in Mondo Cane was staged, based on the stories. ↩
The movie Mondo Cane directly influenced the pop-culture cargo cult as shown by several books. The book River of Tears: The Rise of the Rio Tinto-Zinc Mining Corporation explains cargo cults and how one tribe built an "aeroplane on a hilltop to attract the white man's aeroplane and its cargo", citing Mondo Cane. Likewise, the book Introducing Social Change states that underdeveloped nations are moving directly from ships to airplanes without building railroads, bizarrely using the cargo cult scene in Mondo Cane as an example. Finally, the religious book Open Letter to God uses the cargo cult in Mondo Cane as an example of the suffering of godless people. ↩
Another possibility is that Feynman got his cargo cult ideas from the 1974 book Cows, Pigs, Wars and Witches: The Riddle of Culture. It has a chapter "Phantom Cargo", which starts with a description suspiciously similar to the scene in Mondo Cane:
The scene is a jungle airstrip high in the mountains of New Guinea. Nearby are thatch-roofed hangars, a radio shack, and a beacon tower made of bamboo. On the ground is an airplane made of sticks and leaves. The airstrip is manned twenty-four hours a day by a group of natives wearing nose ornaments and shell armbands. At night they keep a bonfire going to serve as a beacon. They are expecting the arrival of an important flight: cargo planes filled with canned food, clothing, portable radios, wrist watches, and motorcycles. The planes will be piloted by ancestors who have come back to life. Why the delay? A man goes inside the radio shack and gives instructions into the tin-can microphone. The message goes out over an antenna constructed of string and vines: “Do you read me? Roger and out.” From time to time they watch a jet trail crossing the sky; occasionally they hear the sound of distant motors. The ancestors are overhead! They are looking for them. But the whites in the towns below are also sending messages. The ancestors are confused. They land at the wrong airport.↩
Some other uses of the radio telescope photo as a cargo-cult item are Cargo cults, Melanesian cargo cults and the unquenchable thirst of consumerism, Cargo Cult : Correlation vs. Causation, Cargo Cult Agile, Stop looking for silver bullets, and Cargo Cult Investing. ↩
Chariots of the Gods claims to be showing a cargo cult from an isolated island in the South Pacific. However, the large succulent plants in the scene are Euphorbia ingens and tree aloe, which grow in southern Africa, not the South Pacific. The rock formations at the very beginning look a lot like Matobo Hills in Zimbabwe. Note that these "Stone Age" people are astounded by the modern world but ignore the cameraman who is walking among them.
Many cargo cults articles use photos that can be traced back from this film, such as The Scrum Cargo Cult, Is Your UX Cargo Cult, The Remote South Pacific Island Where They Worship Planes, The Design of Everyday Games, Don’t be Fooled by the Bitcoin Core Cargo Cult, The Dying Art of Design, Retail Apocalypse Not, You Are Not Google, and Cargo Cults. The general theme of these articles is that you shouldn't copy what other people are doing without understanding it, which is somewhat ironic. ↩
The Jargon File defined "cargo-cult programming" in 1991:
cargo-cult programming: n. A style of (incompetent) programming dominated by ritual inclusion of code or program structures that serve no real purpose. A cargo-cult programmer will usually explain the extra code as a way of working around some bug encountered in the past, but usually, neither the bug nor the reason the code avoided the bug were ever fully understood.The term cargo-cult is a reference to aboriginal religions that grew up in the South Pacific after World War II. The practices of these cults center on building elaborate mockups of airplanes and military style landing strips in the hope of bringing the return of the god-like airplanes that brought such marvelous cargo during the war. Hackish usage probably derives from Richard Feynman's characterization of certain practices as "cargo-cult science" in `Surely You're Joking, Mr. Feynman'.
This definition of "cargo-cult programming" came from a 1991 Usenet post to alt.folklore.computers, quoting Kent Williams. The definition was added to the much-expanded 1991 Jargon File, which was published as The New Hacker's Dictionary in 1993. ↩
Overuse of the cargo cult metaphor isn't specific to programming, of course. The book Cargo Cult: Strange Stories of Desire from Melanesia and Beyond describes how "cargo cult" has been applied to everything from advertisements, social welfare policy, and shoplifting to the Mormons, Euro Disney, and the state of New Mexico.
This book, by Lamont Linstrom, provides a thorough analysis of writings on cargo cults. It takes a questioning, somewhat trenchant look at these writings, illuminating the development of trends in these writings and the lack of objectivity. I recommend this book to anyone interested in the term "cargo cult" and its history. ↩
Some more things that have been called "cargo cult" on Hacker News: the American worldview, ChatGPT fiction, copy and pasting code, hiring, HR, priorities, psychiatry, quantitative tests, religion, SSRI medication, the tech industry, Uber, and young-earth creationism. ↩
Intel released the powerful Pentium processor in 1993, establishing a long-running brand of high-performance processors.1 The Pentium includes a floating-point unit that can rapidly compute functions such as sines, cosines, logarithms, and exponentials. But how does the Pentium compute these functions? Earlier Intel chips used binary algorithms called CORDIC, but the Pentium switched to polynomials to approximate these transcendental functions much faster. The polynomials have carefully-optimized coefficients that are stored in a special ROM inside the chip's floating-point unit. Even though the Pentium is a complex chip with 3.1 million transistors, it is possible to see these transistors under a microscope and read out these constants. The first part of this post discusses how the floating point constant ROM is implemented in hardware. The second part explains how the Pentium uses these constants to evaluate sin, log, and other functions.
The photo below shows the Pentium's thumbnail-sized silicon die under a microscope. I've labeled the main functional blocks; the floating-point unit is in the lower right. The constant ROM (highlighted) is at the bottom of the floating-point unit. Above the floating-point unit, the microcode ROM holds micro-instructions, the individual steps for complex instructions. To execute an instruction such as sine, the microcode ROM directs the floating-point unit through dozens of steps to compute the approximation polynomial using constants from the constant ROM.
In binary, pi is 11.00100100001111110... but what does this mean?
To interpret this, the value 11 to the left of the binary point is simply 3 in binary. (The "binary point" is the
same as a decimal point, except for binary.)
The digits to the right of the binary point have the values 1/2, 1/4, 1/8, and so forth.
Thus, the binary value `11.001001000011... corresponds to 3 + 1/8 + 1/64 + 1/4096 + 1/8192 + ..., which matches the decimal value of pi.
Since pi is irrational, the bit sequence is infinite and non-repeating; the value in the ROM is truncated to 67 bits
and stored as a floating point number.
A floating point number is represented by two parts: the exponent and the significand. Floating point numbers include very large numbers such as 6.02×1023 and very small numbers such as 1.055×10−34. In decimal, 6.02×1023 has a significand (or mantissa) of 6.02, multiplied by a power of 10 with an exponent of 23. In binary, a floating point number is represented similarly, with a significand and exponent, except the significand is multiplied by a power of 2 rather than 10. For example, pi is represented in floating point as 1.1001001...×21.
The diagram below shows how pi is encoded in the Pentium chip. Zooming in shows the constant ROM. Zooming in on a small part of the ROM shows the rows of transistors that store the constants. The arrows point to the transistors representing the bit sequence 11001001, where a 0 bit is represented by a transistor (vertical white line) and a 1 bit is represented by no transistor (solid dark silicon). Each magnified black rectangle at the bottom has two potential transistors, storing two bits. The key point is that by looking at the pattern of stripes, we can determine the pattern of transistors and thus the value of each constant, pi in this case.
The bits are spread out because each row of the ROM holds eight interleaved constants to improve the layout. Above the ROM bits, multiplexer circuitry selects the desired constant from the eight in the activated row. In other words, by selecting a row and then one of the eight constants in the row, one of the 304 constants in the ROM is accessed. The ROM stores many more digits of pi than shown here; the diagram shows 8 of the 67 significand bits.
The ROM is built from MOS (metal-oxide-semiconductor) transistors, the transistors used in all modern computers. The diagram below shows the structure of an MOS transistor. An integrated circuit is constructed from a silicon substrate. Regions of the silicon are doped with impurities to create "diffusion" regions with desired electrical properties. The transistor can be viewed as a switch, allowing current to flow between two diffusion regions called the source and drain. The transistor is controlled by the gate, made of a special type of silicon called polysilicon. Applying voltage to the gate lets current flow between the source and drain, which is otherwise blocked. Most computers use two types of MOS transistors: NMOS and PMOS. The two types have similar construction but reverse the doping; NMOS uses n-type diffusion regions as shown below, while PMOS uses p-type diffusion regions. Since the two types are complementary (C), circuits built with the two types of transistors are called CMOS.
The image below shows how a transistor in the ROM looks under the microscope. The pinkish regions are the doped silicon that forms the transistor's source and drain. The vertical white line is the polysilicon that forms the transistor's gate. For this photo, I removed the chip's three layers of metal, leaving just the underlying silicon and the polysilicon. The circles in the source and drain are tungsten contacts that connect the silicon to the metal layer above.
The diagram below shows eight bits of storage. Each of the four pink silicon rectangles has two potential transistors. If a polysilicon gate crosses the silicon, a transistor is formed; otherwise there is no transistor. When a select line (horizontal polysilicon) is energized, it will turn on all the transistors in that row. If a transistor is present, the corresponding ROM bit is 0 because the transistor will pull the output line to ground. If a transistor is absent, the ROM bit is 1. Thus, the pattern of transistors determines the data stored in the ROM. The ROM holds 26144 bits (304 words of 86 bits) so it has 26144 potential transistors.
The photo below shows the bottom layer of metal (M1): vertical metal wires that provide the ROM outputs and supply ground to the ROM. (These wires are represented by gray lines in the schematic above.) The polysilicon transistors (or gaps as appropriate) are barely visible between the metal lines. Most of the small circles are tungsten contacts to the silicon or polysilicon; compare with the photo above. Other circles are tungsten vias to the metal layer on top (M2), horizontal wiring that I removed for this photo. The smaller metal "tabs" act as jumpers between the horizontal metal select lines in M2 and the polysilicon select lines. The top metal layer (M3, not visible) has thicker vertical wiring for the chip's primary distribution power and ground. Thus, the three metal layers alternate between horizontal and vertical wiring, with vias between the layers.
The ROM is implemented as two grids of cells (below): one to hold exponents and one to hold significands, as shown below. The exponent grid (on the left) has 38 rows and 144 columns of transistors, while the significand grid (on the right) has 38 rows and 544 columns. To make the layout work better, each row holds eight different constants; the bits are interleaved so the ROM holds the first bit of eight constants, then the second bit of eight constants, and so forth. Thus, with 38 rows, the ROM holds 304 constants; each constant has 18 bits in the exponent part and 68 bits in the significand section.
The exponent part of each constant consists of 18 bits: a 17-bit exponent and one bit for
the sign of the significand and thus the constant.
There is no sign bit for the exponent because
the exponent is stored with 65535 (0x0ffff) added to it, avoiding negative values.
The 68-bit significand entry in the ROM consists of a mysterious flag bit2 followed by the 67-bit significand; the first bit of the significand is the integer part and the remainder is
the fractional part.3
The complete contents of the ROM are in the appendix at the bottom of this post.
To select a particular constant, the "row select" circuitry between the two sections activates one of the 38 rows. That row provides 144+544 bits to the selection circuitry above the ROM. This circuitry has 86 multiplexers; each multiplexer selects one bit out of the group of 8, selecting the desired constant. The significand bits flow into the floating-point unit datapath circuitry above the ROM. The exponent circuitry, however, is in the upper-left corner of the floating-point unit, a considerable distance from the ROM, so the exponent bits travel through a bus to the exponent circuitry.
The row select circuitry consists of gates to decode the row number, along with high-current drivers to energize the selected row in the ROM. The photo below shows a closeup of two row driver circuits, next to some ROM cells. At the left, PMOS and NMOS transistors implement a gate to select the row. Next, larger NMOS and PMOS transistors form part of the driver. The large square structures are bipolar NPN transistors; the Pentium is unusual because it uses both bipolar transistors and CMOS, a technique called BiCMOS.4 Each driver occupies as much height as four rows of the ROM, so there are four drivers arranged horizontally; only one is visible in the photo.
The floating-point unit is structured with data flowing vertically through horizontal functional units, as shown below. The functional units—adders, shifters, registers, and comparators—are arranged in rows. This collection of functional units with data flowing through them is called the datapath.5
Each functional unit is constructed from cells, one per bit, with the high-order bit on the left and the low-order bit on the right. Each cell has the same width—38.5 µm—so the functional units can be connected like Lego blocks snapping together, minimizing the wiring. The height of a functional unit varies as needed, depending on the complexity of the circuit. Functional units typically have 69 bits, but some are wider, so the edges of the datapath circuitry are ragged.
This cell-based construction explains why the ROM has eight constants per row. A ROM bit requires a single transistor, which is much narrower than, say, an adder. Thus, putting one bit in each 38.5 µm cell would waste most of the space. Compacting the ROM bits into a narrow block would also be inefficient, requiring diagonal wiring to connect each ROM bit to the corresponding datapath bit. By putting eight bits for eight different constants into each cell, the width of a ROM cell matches the rest of the datapath and the alignment of bits is preserved. Thus, the layout of the ROM in silicon is dense, efficient, and matches the width of the rest of the floating-point unit.
Now I'll move from the hardware to the constants. If you look at the constant ROM contents in the appendix, you may notice that many constants are close to reciprocals or reciprocal factorials, but don't quite match. For instance, one constant is 0.1111111089, which is close to 1/9, but visibly wrong. Another constant is almost 1/13! (factorial) but wrong by 0.1%. What's going on?
The Pentium uses polynomials to approximate transcendental functions (sine, cosine, tangent, arctangent, and base-2 powers and logarithms). Intel's earlier floating-point units, from the 8087 to the 486, used an algorithm called CORDIC that generated results a bit at a time. However, the Pentium takes advantage of its fast multiplier and larger ROM and uses polynomials instead, computing results two to three times faster than the 486 algorithm.
You may recall from calculus that a Taylor series polynomial approximates a function near a point (typically 0). For example, the equation below gives the Taylor series for sine.
Using the five terms shown above generates a function that looks indistinguishable from sine in the graph below. However, it turns out that this approximation has too much error to be useful.
The problem is that a Taylor series is very accurate near 0, but the error soars near the edges of the argument range, as shown in the graph on the left below. When implementing a function, we want the function to be accurate everywhere, not just close to 0, so the Taylor series isn't good enough.
One improvement is called range reduction: shrinking the argument to a smaller range so you're in the accurate flat part.6 The graph on the right looks at the Taylor series over the smaller range [-1/32, 1/32]. This decreases the error dramatically, by about 22 orders of magnitude (note the scale change). However, the error still shoots up at the edges of the range in exactly the same way. No matter how much you reduce the range, there is almost no error in the middle, but the edges have a lot of error.7
How can we get rid of the error near the edges? The trick is to tweak the coefficients of the Taylor series in a special way that will increase the error in the middle, but decrease the error at the edges by much more. Since we want to minimize the maximum error across the range (called minimax), this tradeoff is beneficial. Specifically, the coefficients can be optimized by a process called the Remez algorithm.8 As shown below, changing the coefficients by less than 1% dramatically improves the accuracy. The optimized function (blue) has much lower error over the full range, so it is a much better approximation than the Taylor series (orange).
To summarize, a Taylor series is useful in calculus, but shouldn't be used to approximate a function. You get a much better approximation by modifying the coefficients very slightly with the Remez algorithm. This explains why the coefficients in the ROM almost, but not quite, match a Taylor series.
I'll now look at the Pentium's constants for different transcendental functions. The constant ROM contains coefficients for two arctan polynomials, one for single precision and one for double precision. These polynomials almost match the Taylor series, but have been modified for accuracy. The ROM also holds the values for arctan(1/32) through arctan(32/32); the range reduction process uses these constants with a trig identity to reduce the argument range to [-1/64, 1/64].9 You can see the arctan constants in the Appendix.
The graph below shows the error for the Pentium's arctan polynomial (blue) versus the Taylor series of the same length (orange). The Pentium's polynomial is superior due to the Remez optimization. Although the Taylor series polynomial is much flatter in the middle, the error soars near the boundary. The Pentium's polynomial wiggles more but it maintains a low error across the whole range. The error in the Pentium polynomial blows up outside this range, but that doesn't matter.
Sine and cosine each have two polynomial implementations, one with 4 terms in the ROM and one with 6 terms in the ROM. (Note that coefficients of 1 are not stored in the ROM.) The constant table also holds 16 constants such as sin(36/64) and cos(18/64) that are used for argument range reduction.10 The Pentium computes tangent by dividing the sine by the cosine. I'm not showing a graph because the Pentium's error came out worse than the Taylor series, so either I have an error in a coefficient or I'm doing something wrong.
The Pentium has an instruction to compute a power of two.11 There are two sets of polynomial coefficients for exponential, one with 6 terms in the ROM and one with 11 terms in the ROM. Curiously, the polynomials in the ROM compute ex, not 2x. Thus, the Pentium must scale the argument by ln(2), a constant that is in the ROM. The error graph below shows the advantage of the Pentium's polynomial over the Taylor series polynomial.
The polynomial handles the narrow argument range [-1/128, 1/128]. Observe that when computing a power of 2 in binary, exponentiating the integer part of the argument is trivial, since it becomes the result's exponent. Thus, the function only needs to handle the range [1, 2]. For range reduction, the constant ROM holds 64 values of the form 2n/128-1. To reduce the range from [1, 2] to [-1/128, 1/128], the closest n/128 is subtracted from the argument and then the result is multiplied by the corresponding constant in the ROM. The constants are spaced irregularly, presumably for accuracy; some are in steps of 4/128 and others are in steps of 2/128.
The Pentium can compute base-2 logarithms.12 The coefficients define polynomials for the hyperbolic arctan, which is closely related to log. See the comments for details. The ROM also has 64 constants for range reduction: log2(1+n/64) for odd n from 1 to 63. The unusual feature of these constants is that each constant is split into two pieces to increase the bits of accuracy: the top part has 40 bits of accuracy and the bottom part has 67 bits of accuracy, providing a 107-bit constant in total. The extra bits are required because logarithms are hard to compute accurately.
The x87 floating-point instruction set provides direct access to a handful of constants—0, 1, pi,
log2(10), log2(e), log10(2), and loge(2)—so these constants
are stored in the ROM.
(These logs are useful for changing the base for logs and exponentials.)
The ROM holds other constants for internal use by the floating-point unit such as -1, 2, 7/8, 9/8, pi/2, pi/4, and 2log2(e).
The ROM also holds bitmasks for extracting part of a word, for instance accessing 4-bit BCD digits in a word.
Although I can interpret most of the values, there are a few mysteries such as a mask with the inscrutable value
0x3e8287c.
The ROM has 34 unused entries at the end; these entries hold words that include the descriptive hex value 0xbad or perhaps 0xbadfc for "bad float constant".
To analyze the Pentium, I removed the metal and oxide layers with various chemicals (sulfuric acid, phosphoric acid, Whink). (I later discovered that simply sanding the die works surprisingly well.) Next, I took many photos of the ROM with a microscope. The feature size of this Pentium is 800 nm, just slightly larger than visible light (380-700 nm). Thus, the die can be examined under an optical microscope, but it is getting close to the limits. To determine the ROM contents, I tediously went through the ROM images, examining each of the 26144 bits and marking each transistor. After figuring out the ROM format, I wrote programs to combine simple functions in many different combinations to determine the mathematical expression such as arctan(19/32) or log2(10). Because the polynomial constants are optimized and my ROM data has bit errors, my program needed checks for inexact matches, both numerically and bitwise. Finally, I had to determine how the constants would be used in algorithms.
By examining the Pentium's floating-point ROM under a microscope, it is possible to extract the 304 constants stored in the ROM. I was able to determine the meaning of most of these constants and deduce some of the floating-point algorithms used by the Pentium. These constants illustrate how polynomials can efficiently compute transcendental functions. Although Taylor series polynomials are well known, they are surprisingly inaccurate and should be avoided. Minor changes to the coefficients through the Remez algorithm, however, yield much better polynomials.
In a previous article, I examined the floating-point constants stored in the 8087 coprocessor. The Pentium has 304 constants in the Pentium, compared to just 42 in the 8087, supporting more efficient algorithms. Moreover, the 8087 was an external floating-point unit, while the Pentium's floating-point unit is part of the processor. The changes between the 8087 (1980, 65,000 transistors) and the Pentium (1993, 3.1 million transistors) are due to the exponential improvements in transistor count, as described by Moore's Law.
I plan to write more about the Pentium so follow me on Bluesky (@righto.com) or RSS for updates. (I'm no longer on Twitter.) I've also written about the Pentium division bug and the Pentium Navajo rug. Thanks to CuriousMarc for microscope help. Thanks to lifthrasiir and Alexia for identifying some constants.
The table below lists the 304 constants in the Pentium's floating-point ROM.
The first four columns show the values stored in the ROM: the exponent, the sign bit, the flag bit, and the
significand.
To avoid negative exponents, exponents are stored with the constant 0x0ffff added. For example, the value 0x0fffe
represents an exponent of -1, while 0x10000 represents an exponent of 1.
The constant's approximate decimal value is in the "value" column.
Special-purpose values are colored. Specifically, "normal" numbers are in black. Constants with an exponent of all 0's are in blue, constants with an exponent of all 1's are in red, constants with an unusually large or small exponent are in green; these appear to be bitmasks rather than numbers. Unused entries are in gray. Inexact constants (due to Remez optimization) are represented with the approximation symbol "≈".
This information is from my reverse engineering, so there will be a few errors.
| exp | S | F | significand | value | meaning | |
|---|---|---|---|---|---|---|
| 0 | 00000 | 0 | 0 | 07878787878787878 | BCD mask by 4's | |
| 1 | 00000 | 0 | 0 | 007f807f807f807f8 | BCD mask by 8's | |
| 2 | 00000 | 0 | 0 | 00007fff80007fff8 | BCD mask by 16's | |
| 3 | 00000 | 0 | 0 | 000000007fffffff8 | BCD mask by 32's | |
| 4 | 00000 | 0 | 0 | 78000000000000000 | 4-bit mask | |
| 5 | 00000 | 0 | 0 | 18000000000000000 | 2-bit mask | |
| 6 | 00000 | 0 | 0 | 27000000000000000 | ? | |
| 7 | 00000 | 0 | 0 | 363c0000000000000 | ? | |
| 8 | 00000 | 0 | 0 | 3e8287c0000000000 | ? | |
| 9 | 00000 | 0 | 0 | 470de4df820000000 | 213×1016 | |
| 10 | 00000 | 0 | 0 | 5c3bd5191b525a249 | 2123/1017 | |
| 11 | 00000 | 0 | 0 | 00000000000000007 | 3-bit mask | |
| 12 | 1ffff | 1 | 1 | 7ffffffffffffffff | all 1's | |
| 13 | 00000 | 0 | 0 | 0000007ffffffffff | mask for 32-bit float | |
| 14 | 00000 | 0 | 0 | 00000000000003fff | mask for 64-bit float | |
| 15 | 00000 | 0 | 0 | 00000000000000000 | all 0's | |
| 16 | 0ffff | 0 | 0 | 40000000000000000 | 1 | 1 |
| 17 | 10000 | 0 | 0 | 6a4d3c25e68dc57f2 | 3.3219280949 | log2(10) |
| 18 | 0ffff | 0 | 0 | 5c551d94ae0bf85de | 1.4426950409 | log2(e) |
| 19 | 10000 | 0 | 0 | 6487ed5110b4611a6 | 3.1415926536 | pi |
| 20 | 0ffff | 0 | 0 | 6487ed5110b4611a6 | 1.5707963268 | pi/2 |
| 21 | 0fffe | 0 | 0 | 6487ed5110b4611a6 | 0.7853981634 | pi/4 |
| 22 | 0fffd | 0 | 0 | 4d104d427de7fbcc5 | 0.3010299957 | log10(2) |
| 23 | 0fffe | 0 | 0 | 58b90bfbe8e7bcd5f | 0.6931471806 | ln(2) |
| 24 | 1ffff | 0 | 0 | 40000000000000000 | +infinity | |
| 25 | 0bfc0 | 0 | 0 | 40000000000000000 | 1/4 of smallest 80-bit denormal? | |
| 26 | 1ffff | 1 | 0 | 60000000000000000 | NaN (not a number) | |
| 27 | 0ffff | 1 | 0 | 40000000000000000 | -1 | -1 |
| 28 | 10000 | 0 | 0 | 40000000000000000 | 2 | 2 |
| 29 | 00000 | 0 | 0 | 00000000000000001 | low bit | |
| 30 | 00000 | 0 | 0 | 00000000000000000 | all 0's | |
| 31 | 00001 | 0 | 0 | 00000000000000000 | single exponent bit | |
| 32 | 0fffe | 0 | 0 | 58b90bfbe8e7bcd5e | 0.6931471806 | ln(2) |
| 33 | 0fffe | 0 | 0 | 40000000000000000 | 0.5 | 1/2! (exp Taylor series) |
| 34 | 0fffc | 0 | 0 | 5555555555555584f | 0.1666666667 | ≈1/3! |
| 35 | 0fffa | 0 | 0 | 555555555397fffd4 | 0.0416666667 | ≈1/4! |
| 36 | 0fff8 | 0 | 0 | 444444444250ced0c | 0.0083333333 | ≈1/5! |
| 37 | 0fff5 | 0 | 0 | 5b05c3dd3901cea50 | 0.0013888934 | ≈1/6! |
| 38 | 0fff2 | 0 | 0 | 6806988938f4f2318 | 0.0001984134 | ≈1/7! |
| 39 | 0fffe | 0 | 0 | 40000000000000000 | 0.5 | 1/2! (exp Taylor series) |
| 40 | 0fffc | 0 | 0 | 5555555555555558e | 0.1666666667 | ≈1/3! |
| 41 | 0fffa | 0 | 0 | 5555555555555558b | 0.0416666667 | ≈1/4! |
| 42 | 0fff8 | 0 | 0 | 444444444443db621 | 0.0083333333 | ≈1/5! |
| 43 | 0fff5 | 0 | 0 | 5b05b05b05afd42f4 | 0.0013888889 | ≈1/6! |
| 44 | 0fff2 | 0 | 0 | 68068068163b44194 | 0.0001984127 | ≈1/7! |
| 45 | 0ffef | 0 | 0 | 6806806815d1b6d8a | 0.0000248016 | ≈1/8! |
| 46 | 0ffec | 0 | 0 | 5c778d8e0384c73ab | 2.755731e-06 | ≈1/9! |
| 47 | 0ffe9 | 0 | 0 | 49f93e0ef41d6086b | 2.755731e-07 | ≈1/10! |
| 48 | 0ffe5 | 0 | 0 | 6ba8b65b40f9c0ce8 | 2.506632e-08 | ≈1/11! |
| 49 | 0ffe2 | 0 | 0 | 47c5b695d0d1289a8 | 2.088849e-09 | ≈1/12! |
| 50 | 0fffd | 0 | 0 | 6dfb23c651a2ef221 | 0.4296133384 | 266/128-1 |
| 51 | 0fffd | 0 | 0 | 75feb564267c8bf6f | 0.4609177942 | 270/128-1 |
| 52 | 0fffd | 0 | 0 | 7e2f336cf4e62105d | 0.4929077283 | 274/128-1 |
| 53 | 0fffe | 0 | 0 | 4346ccda249764072 | 0.5255981507 | 278/128-1 |
| 54 | 0fffe | 0 | 0 | 478d74c8abb9b15cc | 0.5590044002 | 282/128-1 |
| 55 | 0fffe | 0 | 0 | 4bec14fef2727c5cf | 0.5931421513 | 286/128-1 |
| 56 | 0fffe | 0 | 0 | 506333daef2b2594d | 0.6280274219 | 290/128-1 |
| 57 | 0fffe | 0 | 0 | 54f35aabcfedfa1f6 | 0.6636765803 | 294/128-1 |
| 58 | 0fffe | 0 | 0 | 599d15c278afd7b60 | 0.7001063537 | 298/128-1 |
| 59 | 0fffe | 0 | 0 | 5e60f4825e0e9123e | 0.7373338353 | 2102/128-1 |
| 60 | 0fffe | 0 | 0 | 633f8972be8a5a511 | 0.7753764925 | 2106/128-1 |
| 61 | 0fffe | 0 | 0 | 68396a503c4bdc688 | 0.8142521755 | 2110/128-1 |
| 62 | 0fffe | 0 | 0 | 6d4f301ed9942b846 | 0.8539791251 | 2114/128-1 |
| 63 | 0fffe | 0 | 0 | 7281773c59ffb139f | 0.8945759816 | 2118/128-1 |
| 64 | 0fffe | 0 | 0 | 77d0df730ad13bb90 | 0.9360617935 | 2122/128-1 |
| 65 | 0fffe | 0 | 0 | 7d3e0c0cf486c1748 | 0.9784560264 | 2126/128-1 |
| 66 | 0fffc | 0 | 0 | 642e1f899b0626a74 | 0.1956643920 | 233/128-1 |
| 67 | 0fffc | 0 | 0 | 6ad8abf253fe1928c | 0.2086843236 | 235/128-1 |
| 68 | 0fffc | 0 | 0 | 7195cda0bb0cb0b54 | 0.2218460330 | 237/128-1 |
| 69 | 0fffc | 0 | 0 | 7865b862751c90800 | 0.2351510639 | 239/128-1 |
| 70 | 0fffc | 0 | 0 | 7f48a09590037417f | 0.2486009772 | 241/128-1 |
| 71 | 0fffd | 0 | 0 | 431f5d950a896dc70 | 0.2621973504 | 243/128-1 |
| 72 | 0fffd | 0 | 0 | 46a41ed1d00577251 | 0.2759417784 | 245/128-1 |
| 73 | 0fffd | 0 | 0 | 4a32af0d7d3de672e | 0.2898358734 | 247/128-1 |
| 74 | 0fffd | 0 | 0 | 4dcb299fddd0d63b3 | 0.3038812652 | 249/128-1 |
| 75 | 0fffd | 0 | 0 | 516daa2cf6641c113 | 0.3180796013 | 251/128-1 |
| 76 | 0fffd | 0 | 0 | 551a4ca5d920ec52f | 0.3324325471 | 253/128-1 |
| 77 | 0fffd | 0 | 0 | 58d12d497c7fd252c | 0.3469417862 | 255/128-1 |
| 78 | 0fffd | 0 | 0 | 5c9268a5946b701c5 | 0.3616090206 | 257/128-1 |
| 79 | 0fffd | 0 | 0 | 605e1b976dc08b077 | 0.3764359708 | 259/128-1 |
| 80 | 0fffd | 0 | 0 | 6434634ccc31fc770 | 0.3914243758 | 261/128-1 |
| 81 | 0fffd | 0 | 0 | 68155d44ca973081c | 0.4065759938 | 263/128-1 |
| 82 | 0fffd | 1 | 0 | 4cee3bed56eedb76c | -0.3005101637 | 2-66/128-1 |
| 83 | 0fffd | 1 | 0 | 50c4875296f5bc8b2 | -0.3154987885 | 2-70/128-1 |
| 84 | 0fffd | 1 | 0 | 5485c64a56c12cc8a | -0.3301662380 | 2-74/128-1 |
| 85 | 0fffd | 1 | 0 | 58326c4b169aca966 | -0.3445193942 | 2-78/128-1 |
| 86 | 0fffd | 1 | 0 | 5bcaea51f6197f61f | -0.3585649920 | 2-82/128-1 |
| 87 | 0fffd | 1 | 0 | 5f4faef0468eb03de | -0.3723096215 | 2-86/128-1 |
| 88 | 0fffd | 1 | 0 | 62c12658d30048af2 | -0.3857597319 | 2-90/128-1 |
| 89 | 0fffd | 1 | 0 | 661fba6cdf48059b2 | -0.3989216343 | 2-94/128-1 |
| 90 | 0fffd | 1 | 0 | 696bd2c8dfe7a5ffb | -0.4118015042 | 2-98/128-1 |
| 91 | 0fffd | 1 | 0 | 6ca5d4d0ec1916d43 | -0.4244053850 | 2-102/128-1 |
| 92 | 0fffd | 1 | 0 | 6fce23bceb994e239 | -0.4367391907 | 2-106/128-1 |
| 93 | 0fffd | 1 | 0 | 72e520a481a4561a5 | -0.4488087083 | 2-110/128-1 |
| 94 | 0fffd | 1 | 0 | 75eb2a8ab6910265f | -0.4606196011 | 2-114/128-1 |
| 95 | 0fffd | 1 | 0 | 78e09e696172efefc | -0.4721774108 | 2-118/128-1 |
| 96 | 0fffd | 1 | 0 | 7bc5d73c5321bfb9e | -0.4834875605 | 2-122/128-1 |
| 97 | 0fffd | 1 | 0 | 7e9b2e0c43fcf88c8 | -0.4945553570 | 2-126/128-1 |
| 98 | 0fffc | 1 | 0 | 53c94402c0c863f24 | -0.1636449102 | 2-33/128-1 |
| 99 | 0fffc | 1 | 0 | 58661eccf4ca790d2 | -0.1726541162 | 2-35/128-1 |
| 100 | 0fffc | 1 | 0 | 5cf6413b5d2cca73f | -0.1815662751 | 2-37/128-1 |
| 101 | 0fffc | 1 | 0 | 6179ce61cdcdce7db | -0.1903824324 | 2-39/128-1 |
| 102 | 0fffc | 1 | 0 | 65f0e8f35f84645cf | -0.1991036222 | 2-41/128-1 |
| 103 | 0fffc | 1 | 0 | 6a5bb3437adf1164b | -0.2077308674 | 2-43/128-1 |
| 104 | 0fffc | 1 | 0 | 6eba4f46e003a775a | -0.2162651800 | 2-45/128-1 |
| 105 | 0fffc | 1 | 0 | 730cde94abb7410d5 | -0.2247075612 | 2-47/128-1 |
| 106 | 0fffc | 1 | 0 | 775382675996699ad | -0.2330590011 | 2-49/128-1 |
| 107 | 0fffc | 1 | 0 | 7b8e5b9dc385331ad | -0.2413204794 | 2-51/128-1 |
| 108 | 0fffc | 1 | 0 | 7fbd8abc1e5ee49f2 | -0.2494929652 | 2-53/128-1 |
| 109 | 0fffd | 1 | 0 | 41f097f679f66c1db | -0.2575774171 | 2-55/128-1 |
| 110 | 0fffd | 1 | 0 | 43fcb5810d1604f37 | -0.2655747833 | 2-57/128-1 |
| 111 | 0fffd | 1 | 0 | 46032dbad3f462152 | -0.2734860021 | 2-59/128-1 |
| 112 | 0fffd | 1 | 0 | 48041035735be183c | -0.2813120013 | 2-61/128-1 |
| 113 | 0fffd | 1 | 0 | 49ff6c57a12a08945 | -0.2890536989 | 2-63/128-1 |
| 114 | 0fffd | 1 | 0 | 555555555555535f0 | -0.3333333333 | ≈-1/3 (arctan Taylor series) |
| 115 | 0fffc | 0 | 0 | 6666666664208b016 | 0.2 | ≈ 1/5 |
| 116 | 0fffc | 1 | 0 | 492491e0653ac37b8 | -0.1428571307 | ≈-1/7 |
| 117 | 0fffb | 0 | 0 | 71b83f4133889b2f0 | 0.1110544094 | ≈ 1/9 |
| 118 | 0fffd | 1 | 0 | 55555555555555543 | -0.3333333333 | ≈-1/3 (arctan Taylor series) |
| 119 | 0fffc | 0 | 0 | 66666666666616b73 | 0.2 | ≈ 1/5 |
| 120 | 0fffc | 1 | 0 | 4924924920fca4493 | -0.1428571429 | ≈-1/7 |
| 121 | 0fffb | 0 | 0 | 71c71c4be6f662c91 | 0.1111111089 | ≈ 1/9 |
| 122 | 0fffb | 1 | 0 | 5d16e0bde0b12eee8 | -0.0909075848 | ≈-1/11 |
| 123 | 0fffb | 0 | 0 | 4e403be3e3c725aa0 | 0.0764169081 | ≈ 1/13 |
| 124 | 00000 | 0 | 0 | 40000000000000000 | single bit mask | |
| 125 | 0fff9 | 0 | 0 | 7ff556eea5d892a14 | 0.0312398334 | arctan(1/32) |
| 126 | 0fffa | 0 | 0 | 7fd56edcb3f7a71b6 | 0.0624188100 | arctan(2/32) |
| 127 | 0fffb | 0 | 0 | 5fb860980bc43a305 | 0.0934767812 | arctan(3/32) |
| 128 | 0fffb | 0 | 0 | 7f56ea6ab0bdb7196 | 0.1243549945 | arctan(4/32) |
| 129 | 0fffc | 0 | 0 | 4f5bbba31989b161a | 0.1549967419 | arctan(5/32) |
| 130 | 0fffc | 0 | 0 | 5ee5ed2f396c089a4 | 0.1853479500 | arctan(6/32) |
| 131 | 0fffc | 0 | 0 | 6e435d4a498288118 | 0.2153576997 | arctan(7/32) |
| 132 | 0fffc | 0 | 0 | 7d6dd7e4b203758ab | 0.2449786631 | arctan(8/32) |
| 133 | 0fffd | 0 | 0 | 462fd68c2fc5e0986 | 0.2741674511 | arctan(9/32) |
| 134 | 0fffd | 0 | 0 | 4d89dcdc1faf2f34e | 0.3028848684 | arctan(10/32) |
| 135 | 0fffd | 0 | 0 | 54c2b6654735276d5 | 0.3310960767 | arctan(11/32) |
| 136 | 0fffd | 0 | 0 | 5bd86507937bc239c | 0.3587706703 | arctan(12/32) |
| 137 | 0fffd | 0 | 0 | 62c934e5286c95b6d | 0.3858826694 | arctan(13/32) |
| 138 | 0fffd | 0 | 0 | 6993bb0f308ff2db2 | 0.4124104416 | arctan(14/32) |
| 139 | 0fffd | 0 | 0 | 7036d3253b27be33e | 0.4383365599 | arctan(15/32) |
| 140 | 0fffd | 0 | 0 | 76b19c1586ed3da2b | 0.4636476090 | arctan(16/32) |
| 141 | 0fffd | 0 | 0 | 7d03742d50505f2e3 | 0.4883339511 | arctan(17/32) |
| 142 | 0fffe | 0 | 0 | 4195fa536cc33f152 | 0.5123894603 | arctan(18/32) |
| 143 | 0fffe | 0 | 0 | 4495766fef4aa3da8 | 0.5358112380 | arctan(19/32) |
| 144 | 0fffe | 0 | 0 | 47802eaf7bfacfcdb | 0.5585993153 | arctan(20/32) |
| 145 | 0fffe | 0 | 0 | 4a563964c238c37b1 | 0.5807563536 | arctan(21/32) |
| 146 | 0fffe | 0 | 0 | 4d17c07338deed102 | 0.6022873461 | arctan(22/32) |
| 147 | 0fffe | 0 | 0 | 4fc4fee27a5bd0f68 | 0.6231993299 | arctan(23/32) |
| 148 | 0fffe | 0 | 0 | 525e3e8c9a7b84921 | 0.6435011088 | arctan(24/32) |
| 149 | 0fffe | 0 | 0 | 54e3d5ee24187ae45 | 0.6632029927 | arctan(25/32) |
| 150 | 0fffe | 0 | 0 | 5756261c5a6c60401 | 0.6823165549 | arctan(26/32) |
| 151 | 0fffe | 0 | 0 | 59b598e48f821b48b | 0.7008544079 | arctan(27/32) |
| 152 | 0fffe | 0 | 0 | 5c029f15e118cf39e | 0.7188299996 | arctan(28/32) |
| 153 | 0fffe | 0 | 0 | 5e3daef574c579407 | 0.7362574290 | arctan(29/32) |
| 154 | 0fffe | 0 | 0 | 606742dc562933204 | 0.7531512810 | arctan(30/32) |
| 155 | 0fffe | 0 | 0 | 627fd7fd5fc7deaa4 | 0.7695264804 | arctan(31/32) |
| 156 | 0fffe | 0 | 0 | 6487ed5110b4611a6 | 0.7853981634 | arctan(32/32) |
| 157 | 0fffc | 1 | 0 | 55555555555555555 | -0.1666666667 | ≈-1/3! (sin Taylor series) |
| 158 | 0fff8 | 0 | 0 | 44444444444443e35 | 0.0083333333 | ≈ 1/5! |
| 159 | 0fff2 | 1 | 0 | 6806806806773c774 | -0.0001984127 | ≈-1/7! |
| 160 | 0ffec | 0 | 0 | 5c778e94f50956d70 | 2.755732e-06 | ≈ 1/9! |
| 161 | 0ffe5 | 1 | 0 | 6b991122efa0532f0 | -2.505209e-08 | ≈-1/11! |
| 162 | 0ffde | 0 | 0 | 58303f02614d5e4d8 | 1.604139e-10 | ≈ 1/13! |
| 163 | 0fffd | 1 | 0 | 7fffffffffffffffe | -0.5 | ≈-1/2! (cos Taylor series) |
| 164 | 0fffa | 0 | 0 | 55555555555554277 | 0.0416666667 | ≈ 1/4! |
| 165 | 0fff5 | 1 | 0 | 5b05b05b05a18a1ba | -0.0013888889 | ≈-1/6! |
| 166 | 0ffef | 0 | 0 | 680680675b559f2cf | 0.0000248016 | ≈ 1/8! |
| 167 | 0ffe9 | 1 | 0 | 49f93af61f5349300 | -2.755730e-07 | ≈-1/10! |
| 168 | 0ffe2 | 0 | 0 | 47a4f2483514c1af8 | 2.085124e-09 | ≈ 1/12! |
| 169 | 0fffc | 1 | 0 | 55555555555555445 | -0.1666666667 | ≈-1/3! (sin Taylor series) |
| 170 | 0fff8 | 0 | 0 | 44444444443a3fdb6 | 0.0083333333 | ≈ 1/5! |
| 171 | 0fff2 | 1 | 0 | 68068060b2044e9ae | -0.0001984127 | ≈-1/7! |
| 172 | 0ffec | 0 | 0 | 5d75716e60f321240 | 2.785288e-06 | ≈ 1/9! |
| 173 | 0fffd | 1 | 0 | 7fffffffffffffa28 | -0.5 | ≈-1/2! (cos Taylor series) |
| 174 | 0fffa | 0 | 0 | 555555555539cfae6 | 0.0416666667 | ≈ 1/4! |
| 175 | 0fff5 | 1 | 0 | 5b05b050f31b2e713 | -0.0013888889 | ≈-1/6! |
| 176 | 0ffef | 0 | 0 | 6803988d56e3bff10 | 0.0000247989 | ≈ 1/8! |
| 177 | 0fffe | 0 | 0 | 44434312da70edd92 | 0.5333026735 | sin(36/64) |
| 178 | 0fffe | 0 | 0 | 513ace073ce1aac13 | 0.6346070800 | sin(44/64) |
| 179 | 0fffe | 0 | 0 | 5cedda037a95df6ee | 0.7260086553 | sin(52/64) |
| 180 | 0fffe | 0 | 0 | 672daa6ef3992b586 | 0.8060811083 | sin(60/64) |
| 181 | 0fffd | 0 | 0 | 470df5931ae1d9460 | 0.2775567516 | sin(18/64) |
| 182 | 0fffd | 0 | 0 | 5646f27e8bd65cbe4 | 0.3370200690 | sin(22/64) |
| 183 | 0fffd | 0 | 0 | 6529afa7d51b12963 | 0.3951673302 | sin(26/64) |
| 184 | 0fffd | 0 | 0 | 73a74b8f52947b682 | 0.4517714715 | sin(30/64) |
| 185 | 0fffe | 0 | 0 | 6c4741058a93188ef | 0.8459244992 | cos(36/64) |
| 186 | 0fffe | 0 | 0 | 62ec41e9772401864 | 0.7728350058 | cos(44/64) |
| 187 | 0fffe | 0 | 0 | 5806149bd58f7d46d | 0.6876855622 | cos(52/64) |
| 188 | 0fffe | 0 | 0 | 4bc044c9908390c72 | 0.5918050751 | cos(60/64) |
| 189 | 0fffe | 0 | 0 | 7af8853ddbbe9ffd0 | 0.9607092430 | cos(18/64) |
| 190 | 0fffe | 0 | 0 | 7882fd26b35b03d34 | 0.9414974631 | cos(22/64) |
| 191 | 0fffe | 0 | 0 | 7594fc1cf900fe89e | 0.9186091558 | cos(26/64) |
| 192 | 0fffe | 0 | 0 | 72316fe3386a10d5a | 0.8921336994 | cos(30/64) |
| 193 | 0ffff | 0 | 0 | 48000000000000000 | 1.125 | 9/8 |
| 194 | 0fffe | 0 | 0 | 70000000000000000 | 0.875 | 7/8 |
| 195 | 0ffff | 0 | 0 | 5c551d94ae0bf85de | 1.4426950409 | log2(e) |
| 196 | 10000 | 0 | 0 | 5c551d94ae0bf85de | 2.8853900818 | 2log2(e) |
| 197 | 0fffb | 0 | 0 | 7b1c2770e81287c11 | 0.1202245867 | ≈1/(41⋅3⋅ln(2)) (atanh series for log) |
| 198 | 0fff9 | 0 | 0 | 49ddb14064a5d30bd | 0.0180336880 | ≈1/(42⋅5⋅ln(2)) |
| 199 | 0fff6 | 0 | 0 | 698879b87934f12e0 | 0.0032206148 | ≈1/(43⋅7⋅ln(2)) |
| 200 | 0fffa | 0 | 0 | 51ff4ffeb20ed1749 | 0.0400377512 | ≈(ln(2)/2)2/3 (atanh series for log) |
| 201 | 0fff6 | 0 | 0 | 5e8cd07eb1827434a | 0.0028854387 | ≈(ln(2)/2)4/5 |
| 202 | 0fff3 | 0 | 0 | 40e54061b26dd6dc2 | 0.0002475567 | ≈(ln(2)/2)6/7 |
| 203 | 0ffef | 0 | 0 | 61008a69627c92fb9 | 0.0000231271 | ≈(ln(2)/2)8/9 |
| 204 | 0ffec | 0 | 0 | 4c41e6ced287a2468 | 2.272648e-06 | ≈(ln(2)/2)10/11 |
| 205 | 0ffe8 | 0 | 0 | 7dadd4ea3c3fee620 | 2.340954e-07 | ≈(ln(2)/2)12/13 |
| 206 | 0fff9 | 0 | 0 | 5b9e5a170b8000000 | 0.0223678130 | log2(1+1/64) top bits |
| 207 | 0fffb | 0 | 0 | 43ace37e8a8000000 | 0.0660892054 | log2(1+3/64) top bits |
| 208 | 0fffb | 0 | 0 | 6f210902b68000000 | 0.1085244568 | log2(1+5/64) top bits |
| 209 | 0fffc | 0 | 0 | 4caba789e28000000 | 0.1497471195 | log2(1+7/64) top bits |
| 210 | 0fffc | 0 | 0 | 6130af40bc0000000 | 0.1898245589 | log2(1+9/64) top bits |
| 211 | 0fffc | 0 | 0 | 7527b930c98000000 | 0.2288186905 | log2(1+11/64) top bits |
| 212 | 0fffd | 0 | 0 | 444c1f6b4c0000000 | 0.2667865407 | log2(1+13/64) top bits |
| 213 | 0fffd | 0 | 0 | 4dc4933a930000000 | 0.3037807482 | log2(1+15/64) top bits |
| 214 | 0fffd | 0 | 0 | 570068e7ef8000000 | 0.3398500029 | log2(1+17/64) top bits |
| 215 | 0fffd | 0 | 0 | 6002958c588000000 | 0.3750394313 | log2(1+19/64) top bits |
| 216 | 0fffd | 0 | 0 | 68cdd829fd8000000 | 0.4093909361 | log2(1+21/64) top bits |
| 217 | 0fffd | 0 | 0 | 7164beb4a58000000 | 0.4429434958 | log2(1+23/64) top bits |
| 218 | 0fffd | 0 | 0 | 79c9aa879d8000000 | 0.4757334310 | log2(1+25/64) top bits |
| 219 | 0fffe | 0 | 0 | 40ff6a2e5e8000000 | 0.5077946402 | log2(1+27/64) top bits |
| 220 | 0fffe | 0 | 0 | 450327ea878000000 | 0.5391588111 | log2(1+29/64) top bits |
| 221 | 0fffe | 0 | 0 | 48f107509c8000000 | 0.5698556083 | log2(1+31/64) top bits |
| 222 | 0fffe | 0 | 0 | 4cc9f1aad28000000 | 0.5999128422 | log2(1+33/64) top bits |
| 223 | 0fffe | 0 | 0 | 508ec1fa618000000 | 0.6293566201 | log2(1+35/64) top bits |
| 224 | 0fffe | 0 | 0 | 5440461c228000000 | 0.6582114828 | log2(1+37/64) top bits |
| 225 | 0fffe | 0 | 0 | 57df3fd0780000000 | 0.6865005272 | log2(1+39/64) top bits |
| 226 | 0fffe | 0 | 0 | 5b6c65a9d88000000 | 0.7142455177 | log2(1+41/64) top bits |
| 227 | 0fffe | 0 | 0 | 5ee863e4d40000000 | 0.7414669864 | log2(1+43/64) top bits |
| 228 | 0fffe | 0 | 0 | 6253dd2c1b8000000 | 0.7681843248 | log2(1+45/64) top bits |
| 229 | 0fffe | 0 | 0 | 65af6b4ab30000000 | 0.7944158664 | log2(1+47/64) top bits |
| 230 | 0fffe | 0 | 0 | 68fb9fce388000000 | 0.8201789624 | log2(1+49/64) top bits |
| 231 | 0fffe | 0 | 0 | 6c39049af30000000 | 0.8454900509 | log2(1+51/64) top bits |
| 232 | 0fffe | 0 | 0 | 6f681c731a0000000 | 0.8703647196 | log2(1+53/64) top bits |
| 233 | 0fffe | 0 | 0 | 72896372a50000000 | 0.8948177633 | log2(1+55/64) top bits |
| 234 | 0fffe | 0 | 0 | 759d4f80cb8000000 | 0.9188632373 | log2(1+57/64) top bits |
| 235 | 0fffe | 0 | 0 | 78a450b8380000000 | 0.9425145053 | log2(1+59/64) top bits |
| 236 | 0fffe | 0 | 0 | 7b9ed1c6ce8000000 | 0.9657842847 | log2(1+61/64) top bits |
| 237 | 0fffe | 0 | 0 | 7e8d3845df0000000 | 0.9886846868 | log2(1+63/64) top bits |
| 238 | 0ffd0 | 1 | 0 | 6eb3ac8ec0ef73f7b | -1.229037e-14 | log2(1+1/64) bottom bits |
| 239 | 0ffcd | 1 | 0 | 654c308b454666de9 | -1.405787e-15 | log2(1+3/64) bottom bits |
| 240 | 0ffd2 | 0 | 0 | 5dd31d962d3728cbd | 4.166652e-14 | log2(1+5/64) bottom bits |
| 241 | 0ffd3 | 0 | 0 | 70d0fa8f9603ad3a6 | 1.002010e-13 | log2(1+7/64) bottom bits |
| 242 | 0ffd1 | 0 | 0 | 765fba4491dcec753 | 2.628429e-14 | log2(1+9/64) bottom bits |
| 243 | 0ffd2 | 1 | 0 | 690370b4a9afdc5fb | -4.663533e-14 | log2(1+11/64) bottom bits |
| 244 | 0ffd4 | 0 | 0 | 5bae584b82d3cad27 | 1.628582e-13 | log2(1+13/64) bottom bits |
| 245 | 0ffd4 | 0 | 0 | 6f66cc899b64303f7 | 1.978889e-13 | log2(1+15/64) bottom bits |
| 246 | 0ffd4 | 1 | 0 | 4bc302ffa76fafcba | -1.345799e-13 | log2(1+17/64) bottom bits |
| 247 | 0ffd2 | 1 | 0 | 7579aa293ec16410a | -5.216949e-14 | log2(1+19/64) bottom bits |
| 248 | 0ffcf | 0 | 0 | 509d7c40d7979ec5b | 4.475041e-15 | log2(1+21/64) bottom bits |
| 249 | 0ffd3 | 1 | 0 | 4a981811ab5110ccf | -6.625289e-14 | log2(1+23/64) bottom bits |
| 250 | 0ffd4 | 1 | 0 | 596f9d730f685c776 | -1.588702e-13 | log2(1+25/64) bottom bits |
| 251 | 0ffd4 | 1 | 0 | 680cc6bcb9bfa9853 | -1.848298e-13 | log2(1+27/64) bottom bits |
| 252 | 0ffd4 | 0 | 0 | 5439e15a52a31604a | 1.496156e-13 | log2(1+29/64) bottom bits |
| 253 | 0ffd4 | 0 | 0 | 7c8080ecc61a98814 | 2.211599e-13 | log2(1+31/64) bottom bits |
| 254 | 0ffd3 | 1 | 0 | 6b26f28dbf40b7bc0 | -9.517022e-14 | log2(1+33/64) bottom bits |
| 255 | 0ffd5 | 0 | 0 | 554b383b0e8a55627 | 3.030245e-13 | log2(1+35/64) bottom bits |
| 256 | 0ffd5 | 0 | 0 | 47c6ef4a49bc59135 | 2.550034e-13 | log2(1+37/64) bottom bits |
| 257 | 0ffd5 | 0 | 0 | 4d75c658d602e66b0 | 2.751934e-13 | log2(1+39/64) bottom bits |
| 258 | 0ffd4 | 1 | 0 | 6b626820f81ca95da | -1.907530e-13 | log2(1+41/64) bottom bits |
| 259 | 0ffd3 | 0 | 0 | 5c833d56efe4338fe | 8.216774e-14 | log2(1+43/64) bottom bits |
| 260 | 0ffd5 | 0 | 0 | 7c5a0375163ec8d56 | 4.417857e-13 | log2(1+45/64) bottom bits |
| 261 | 0ffd5 | 1 | 0 | 5050809db75675c90 | -2.853343e-13 | log2(1+47/64) bottom bits |
| 262 | 0ffd4 | 1 | 0 | 7e12f8672e55de96c | -2.239526e-13 | log2(1+49/64) bottom bits |
| 263 | 0ffd5 | 0 | 0 | 435ebd376a70d849b | 2.393466e-13 | log2(1+51/64) bottom bits |
| 264 | 0ffd2 | 1 | 0 | 6492ba487dfb264b3 | -4.466345e-14 | log2(1+53/64) bottom bits |
| 265 | 0ffd5 | 1 | 0 | 674e5008e379faa7c | -3.670163e-13 | log2(1+55/64) bottom bits |
| 266 | 0ffd5 | 0 | 0 | 5077f1f5f0cc82aab | 2.858817e-13 | log2(1+57/64) bottom bits |
| 267 | 0ffd2 | 0 | 0 | 5007eeaa99f8ef14d | 3.554090e-14 | log2(1+59/64) bottom bits |
| 268 | 0ffd5 | 0 | 0 | 4a83eb6e0f93f7a64 | 2.647316e-13 | log2(1+61/64) bottom bits |
| 269 | 0ffd3 | 0 | 0 | 466c525173dae9cf5 | 6.254831e-14 | log2(1+63/64) bottom bits |
| 270 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 271 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 272 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 273 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 274 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 275 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 276 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 277 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 278 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 279 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 280 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 281 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 282 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 283 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 284 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 285 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 286 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 287 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 288 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 289 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 290 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 291 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 292 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 293 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 294 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 295 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 296 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 297 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 298 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 299 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 300 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 301 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 302 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused | |
| 303 | 0badf | 0 | 1 | 40badfc0badfc0bad | unused |
In this blog post, I'm looking at the "P5" version of the original Pentium processor. It can be hard to keep all the Pentiums straight since "Pentium" became a brand name with multiple microarchitectures, lines, and products. The original Pentium (1993) was followed by the Pentium Pro (1995), Pentium II (1997), and so on.
The original Pentium used the P5 microarchitecture, a superscalar microarchitecture that was advanced but still executed instruction in order like traditional microprocessors. The original Pentium went through several substantial revisions. The first Pentium product was the 80501 (codenamed P5), containing 3.1 million transistors. The power consumption of these chips was disappointing, so Intel improved the chip, producing the 80502, codenamed P54C. The P5 and P54C look almost the same on the die, but the P54C added circuitry for multiprocessing, boosting the transistor count to 3.3 million. The biggest change to the original Pentium was the Pentium MMX, with part number 80503 and codename P55C. The Pentium MMX added 57 vector processing instructions and had 4.5 million transistors. The floating-point unit was rearranged in the MMX, but the constants are probably the same. ↩
I don't know what the flag bit in the ROM indicates; I'm arbitrarily calling it a flag. My wild guess is that it indicates ROM entries that should be excluded from the checksum when testing the ROM. ↩
Internally, the significand has one integer bit and the remainder is the fraction, so the binary point (decimal point) is after the first bit. However, this is not the only way to represent the significand. The x87 80-bit floating-point format (double extended-precision) uses the same approach. However, the 32-bit (single-precision) and 64-bit (double-precision) formats drop the first bit and use an "implied" one bit. This gives you one more bit of significand "for free" since in normal cases the first significand bit will be 1. ↩
An unusual feature of the Pentium is that it uses bipolar NPN transistors along with CMOS circuits, a technology called BiCMOS. By adding a few extra processing steps to the regular CMOS manufacturing process, bipolar transistors could be created. The Pentium uses BiCMOS circuits extensively since they reduced signal delays by up to 35%. Intel also used BiCMOS for the Pentium Pro, Pentium II, Pentium III, and Xeon processors (but not the Pentium MMX). However, as chip voltages dropped, the benefit from bipolar transistors dropped too and BiCMOS was eventually abandoned.
In the constant ROM, BiCMOS circuits improve the performance of the row selection circuitry. Each row select line is very long and is connected to hundreds of transistors, so the capacitive load is large. Because of the fast and powerful NPN transistor, a BiCMOS driver provides lower delay for higher loads than a regular CMOS driver.
This BiCMOS logic is also called BiNMOS or BinMOS because the output has a bipolar transistor and an NMOS transistor. For more on BiCMOS circuits in the Pentium, see my article Standard cells: Looking at individual gates in the Pentium processor. ↩
The integer processing unit of the Pentium is constructed similarly, with horizontal functional units stacked to form the datapath. Each cell in the integer unit is much wider than a floating-point cell (64 µm vs 38.5 µm). However, the integer unit is just 32 bits wide, compared to 69 (more or less) for the floating-point unit, so the floating-point unit is wider overall. ↩
I don't like referring to the argument's range since a function's output is the range, while its input is the domain. But the term range reduction is what people use, so I'll go with it. ↩
There's a reason why the error curve looks similar even if you reduce the range. The error from the Taylor series is approximately the next term in the Taylor series, so in this case the error is roughly -x11/11! or O(x11). This shows why range reduction is so powerful: if you reduce the range by a factor of 2, you reduce the error by the enormous factor of 211. But this also shows why the error curve keeps its shape: the curve is still x11, just with different labels on the axes. ↩
The Pentium coefficients are probably obtained using the Remez algorithm; see Floating-Point Verification. The advantages of the Remez polynomial over the Taylor series are discussed in Better Function Approximations: Taylor vs. Remez. A description of Remez's algorithm is in Elementary Functions: Algorithms and Implementation, which has other relevant information on polynomial approximation and range reduction. For more on polynomial approximations, see Numerically Computing the Exponential Function with Polynomial Approximations and The Eight Useful Polynomial Approximations of Sinf(3),
The Remez polynomial in the sine graph is not the Pentium polynomial; it was generated for illustration by lolremez, a useful tool. The specific polynomial is:
9.9997938808335731e-1 ⋅ x - 1.6662438518867169e-1 ⋅ x3 + 8.3089850302282266e-3 ⋅ x5 - 1.9264997445395096e-4 ⋅ x7 + 2.1478735041839789e-6 ⋅ x9
The graph below shows the error for this polynomial. Note that the error oscillates between an upper bound and a lower bound. This is the typical appearance of a Remez polynomial. In contrast, a Taylor series will have almost no error in the middle and shoot up at the edges. This Remez polynomial was optimized for the range [-π,π]; the error explodes outside that range. The key point is that the Remez polynomial distributes the error inside the range. This minimizes the maximum error (minimax).
I think the arctan argument is range-reduced to the range [-1/64, 1/64].
This can be accomplished with the trig identity arctan(x) = arctan((x-c)/(1+xc)) + arctan(c).
The idea is that c is selected to be the value of the form n/32 closest to x.
As a result, x-c will be in the desired range and the first arctan can be computed with the polynomial.
The other term, arctan(c), is obtained from the lookup table in the ROM.
The FPATAN (partial arctangent) instruction takes two arguments, x and y, and returns atan(y/x); this simplifies
handling planar coordinates.
In this case, the trig identity becomes arcan(y/x) = arctan((y-tx)/(x+ty)) + arctan c.
The division operation can trigger the FDIV bug in some cases;
see Computational Aspects of the Pentium Affair. ↩
The Pentium has several trig instructions: FSIN, FCOS, and FSINCOS return the sine, cosine, or both
(which is almost as fast as computing either).
FPTAN returns the "partial tangent" consisting of two numbers that must be divided to yield the tangent.
(This was due to limitations in the original 8087 coprocessor.)
The Pentium returns the tangent as the first number and the constant 1 as the second number, keeping the
semantics of FPTAN while being more convenient.
The range reduction is probably based on the trig identity sin(a+b) = sin(a)cos(b)+cos(a)sin(b). To compute sin(x), select b as the closest constant in the lookup table, n/64, and then generate a=x-b. The value a will be range-reduced, so sin(a) can be computed from the polynomial. The terms sin(b) and cos(b) are available from the lookup table. The desired value sin(x) can then be computed with multiplications and addition by using the trig identity. Cosine can be computed similarly. Note that cos(a+b) =cos(a)cos(b)-sin(a)sin(b); the terms on the right are the same as for sin(a+b), just combined differently. Thus, once the terms on the right have been computed, they can be combined to generate sine, cosine, or both. The Pentium computes the tangent by dividing the sine by the cosine. This can trigger the FDIV division bug; see Computational Aspects of the Pentium Affair.
Also see Agner Fog's Instruction Timings;
the timings for the various operations give clues as to how they are computed. For instance, FPTAN takes
longer than FSINCOS because the tangent is generated by dividing the sine by the cosine. ↩
For exponentials, the F2XM1 instruction computes 2x-1;
subtracting 1 improves accuracy.
Specifically, 2x is close to 1 for the common case when x is close to 0, so subtracting 1
as a separate operation causes you to lose most of the bits of accuracy due to cancellation.
On the other hand, if you want 2x, explicitly adding 1 doesn't harm accuracy.
This is an example of how the floating-point instructions are carefully designed to preserve accuracy.
For details,
see the book The 8087 Primer by the architects of the 8086 processor and the 8087 coprocessor. ↩
The Pentium has base-two logarithm instructions FYL2X and FYL2XP1.
The FYL2X instruction computes y log2(x)
and the FYL2XP1 instruction computes y log2(x+1)
The instructions include a multiplication because most logarithm operations will need to multiply to
change the base; performing the multiply with internal precision increases the accuracy.
The "plus-one" instruction improves accuracy for arguments close to 1, such as interest calculations.
My hypothesis for range reduction is that the input argument is scaled to fall between 1 and 2. (Taking the log of the exponent part of the argument is trivial since the base-2 log of a base-2 power is simply the exponent.) The argument can then be divided by the largest constant 1+n/64 less than the argument. This will reduce the argument to the range [1, 1+1/32]. The log polynomial can be evaluated on the reduced argument. Finally, the ROM constant for log2(1+n/64) is added to counteract the division. The constant is split into two parts for greater accuracy.
It took me a long time to figure out the log constants because they were split. The upper-part constants appeared to be pointlessly inaccurate since the bottom 27 bits are zeroed out. The lower-part constants appeared to be miniscule semi-random numbers around ±10-13. Eventually, I figured out that the trick was to combine the constants. ↩
In 1993, Intel released the high-performance Pentium processor, the start of the long-running Pentium line. The Pentium had many improvements over the previous processor, the Intel 486, including a faster floating-point division algorithm. A year later, Professor Nicely, a number theory professor, was researching reciprocals of twin prime numbers when he noticed a problem: his Pentium sometimes generated the wrong result when performing floating-point division. Intel considered this "an extremely minor technical problem", but much to Intel's surprise, the bug became a large media story. After weeks of criticism, mockery, and bad publicity, Intel agreed to replace everyone's faulty Pentium chips, costing the company $475 million.
In this article, I discuss the Pentium's division algorithm, show exactly where the bug is on the Pentium chip, take a close look at the circuitry, and explain what went wrong. In brief, the division algorithm uses a lookup table. In 1994, Intel stated that the cause of the bug was that five entries were omitted from the table due to an error in a script. However, my analysis shows that 16 entries were omitted due to a mathematical mistake in the definition of the lookup table. Five of the missing entries trigger the bug— also called the FDIV bug after the floating-point division instruction "FDIV"—while 11 of the missing entries have no effect.
Although Professor Nicely brought attention to the FDIV bug, he wasn't the first to find it. In May 1994, Intel's internal testing of the Pentium revealed that very rarely, floating-point division was slightly inaccurate.1 Since only one in 9 billion values caused the problem, Intel's view was that the problem was trivial: "This doesn't even qualify as an errata." Nonetheless, Intel quietly revised the Pentium circuitry to fix the problem.
A few months later, in October, Nicely noticed erroneous results in his prime number computations.2 He soon determined that 1/824633702441 was wrong on three different Pentium computers, but his older computers gave the right answer. He called Intel tech support but was brushed off, so Nicely emailed a dozen computer magazines and individuals about the bug. One of the recipients was Andrew Schulman, author of "Undocumented DOS". He forwarded the email to Richard Smith, cofounder of a DOS software tools company. Smith posted the email on a Compuserve forum, a 1990s version of social media.
A reporter for the journal Electronic Engineering Times spotted the Compuserve post and wrote about the Pentium bug in the November 7 issue: Intel fixes a Pentium FPU glitch. In the article, Intel explained that the bug was in a component of the chip called a PLA (Programmable Logic Array) that acted as a lookup table for the division operation. Intel had fixed the bug in the latest Pentiums and would replace faulty processors for concerned customers.3
The problem might have quietly ended here, except that Intel decided to restrict which customers could get a replacement. If a customer couldn't convince an Intel engineer that they needed the accuracy, they couldn't get a fixed Pentium. Users were irate to be stuck with faulty chips so they took their complaints to online groups such as comp.sys.intel. The controversy spilled over into the offline world on November 22 when CNN reported on the bug. Public awareness of the Pentium bug took off as newspapers wrote about the bug and Intel became a punchline on talk shows.4
The situation became intolerable for Intel on December 12 when IBM announced that it was stopping shipments of Pentium computers.5 On December 19, less than two months after Nicely first reported the bug, Intel gave in and announced that it would replace the flawed chips for all customers.6 This recall cost Intel $475 million (over a billion dollars in current dollars).
Meanwhile, engineers and mathematicians were analyzing the bug, including Tim Coe, an engineer who had designed floating-point units.7 Remarkably, by studying the Pentium's bad divisions, Coe reverse-engineered the Pentium's division algorithm and determined why it went wrong. Coe and others wrote papers describing the mathematics behind the Pentium bug.8 But until now, nobody has shown how the bug is implemented in the physical chip itself.
At this point, I'll review a few important things about floating point numbers. A binary number can have a fractional part, similar to a decimal number. For instance, the binary number 11.1001 has four digits after the binary point. (The binary point "." is similar to the decimal point, but for a binary number.) The first digit after the binary point represents 1/2, the second represents 1/4, and so forth. Thus, 11.1001 corresponds to 3 + 1/2 + 1/16 = 3.5625. A "fixed point" number such as this can express a fractional value, but its range is limited.
Floating point numbers, on the other hand, include very large numbers such as 6.02×1023 and very small numbers such as 1.055×10−34. In decimal, 6.02×1023 has a significand (or mantissa) of 6.02, multiplied by a power of 10 with an exponent of 23. In binary, a floating point number is represented similarly, with a significand and exponent, except the significand is multiplied by a power of 2 rather than 10.
Computers have used floating point since the early days of computing, especially for scientific computing. For many years, different computers used incompatible formats for floating point numbers. Eventually, a standard arose when Intel developed the 8087 floating point coprocessor chip for use with the 8086/8088 processor. The characteristics of this chip became a standard (IEEE 754) in 1985.9 Subsequently, most computers, including the Pentium, implemented floating point numbers according to this standard. The result of a basic arithmetic operation is supposed to be accurate up to the last bit of the significand. Unfortunately, division on the Pentium was occasionally much, much worse.
How does a computer perform division? The straightforward way is similar to grade-school long division, except in binary. That approach was used in the Intel 486 and earlier processors, but the process is slow, taking one clock cycle for each bit of the quotient. The Pentium uses a different approach called SRT,10 performing division in base four. Thus, SRT generates two bits of the quotient per step, rather than one, so division is twice as fast. I'll explain SRT in a hand-waving manner with a base-10 example; rigorous explanations are available elsewhere.
The diagram below shows base-10 long division, with the important parts named. The dividend is divided by the divisor, yielding the quotient. In each step of the long division algorithm, you generate one more digit of the quotient. Then you multiply the divisor (1535) by the quotient digit (2) and subtract this from the dividend, leaving a partial remainder. You multiply the partial remainder by 10 and then repeat the process, generating a quotient digit and partial remainder at each step. The diagram below stops after two quotient digits, but you can keep going to get as much accuracy as desired.
Note that division is more difficult than multiplication since there is no easy way to determine each quotient digit. You have to estimate a quotient digit, multiply it by the divisor, and then check if the quotient digit is correct. For example, you have to check carefully to see if 1535 goes into 4578 two times or three times.
The SRT algorithm makes it easier to select the quotient digit through an unusual approach: it allows negative digits in the quotient. With this change, the quotient digit does not need to be exact. If you pick a quotient digit that is a bit too large, you can use a negative number for the next digit: this will counteract the too-large digit since the next divisor will be added rather than subtracted.
The example below shows how this works. Suppose you picked 3 instead of 2 as the first quotient digit. Since 3 is too big, the partial remainder is negative (-261). In normal division, you'd need to try again with a different quotient digit. But with SRT, you keep going, using a negative digit (-1) for the quotient digit in the next step. At the end, the quotient with positive and negative digits can be converted to the standard form: 3×10-1 = 29, the same quotient as before.
One nice thing about the SRT algorithm is that since the quotient digit only needs to be close, a lookup table can be used to select the quotient digit. Specifically, the partial remainder and divisor can be truncated to a few digits, making the lookup table a practical size. In this example, you could truncate 1535 and 4578 to 15 and 45, the table says that 15 goes into 45 three times, and you can use 3 as your quotient digit.
Instead of base 10, the Pentium uses the SRT algorithm in base 4: groups of two bits. As a result, division on the Pentium is twice as fast as standard binary division. With base-4 SRT, each quotient digit can be -2, -1, 0, 1, or 2. Multiplying by any of these values is very easy in hardware since multiplying by 2 can be done by a bit shift. Base-4 SRT does not require quotient digits of -3 or 3; this is convenient since multiplying by 3 is somewhat difficult. To summarize, base-4 SRT is twice as fast as regular binary division, but it requires more hardware: a lookup table, circuitry to add or subtract multiples of 1 or 2, and circuitry to convert the quotient to the standard form.
The purpose of the SRT lookup table is to provide the quotient digit. That is, the table takes the partial remainder p and the divisor d as inputs and provides an appropriate quotient digit. The Pentium's lookup table is the cause of the division bug, as was explained in 1994. The table was missing five entries; if the SRT algorithm accesses one of these missing entries, it generates an incorrect result. In this section, I'll discuss the structure of the lookup table and explain what went wrong.
The Pentium's lookup table contains 2048 entries, as shown below. The table has five regions corresponding to the quotient digits +2, +1, 0, -1, and -2. Moreover, the upper and lower regions of the table are unused (due to the mathematics of SRT). The unused entries were filled with 0, which turns out to be very important. In particular, the five red entries need to contain +2 but were erroneously filled with 0.
When the SRT algorithm uses the table, the partial remainder p and the divisor d are inputs. The divisor (scaled to fall between 1 and 2) provides the X coordinate into the table, while the partial remainder (between -8 and 8) provides the Y coordinate. The details of the table coordinates will be important, so I'll go into some detail. To select a cell, the divisor (X-axis) is truncated to a 5-bit binary value 1.dddd. (Since the first digit of the divisor is always 1, it is ignored for the table lookup.) The partial remainder (Y-axis) is truncated to a 7-bit signed binary value pppp.ppp. The 11 bits indexing into the table result in a table with 211 (2048) entries. The partial remainder is expressed in 2's complement, so values 0000.000 to 0111.111 are non-negative values from 0 to (almost) 8, while values 1000.000 to 1111.111 are negative values from -8 to (almost) 0. (To see the binary coordinates for the table, click on the image and zoom in.)
In this section, I'll explain how the lookup table is implemented in hardware in the Pentium. The lookup table has 2048 entries so it could be stored in a ROM with 2048 two-bit outputs.11 (The sign is not explicitly stored in the table because the quotient digit sign is the same as the partial remainder sign.) However, because the table is highly structured (and largely empty), the table can be stored more compactly in a structure called a Programmable Logic Array (PLA).12 By using a PLA, the Pentium stored the table in just 112 rows rather than 2048 rows, saving an enormous amount of space. Even so, the PLA is large enough on the chip that it is visible to the naked eye, if you squint a bit.
The idea of a PLA is to provide a dense and flexible way of implementing arbitrary logic functions. Any Boolean logic function can be expressed as a "sum-of-products", a collection of AND terms (products) that are OR'd together (summed). A PLA has a block of circuitry called the AND plane that generates the desired sum terms. The outputs of the AND plane are fed into a second block, the OR plane, which ORs the terms together. The AND plane and the OR plane are organized as grids. Each gridpoint can either have a transistor or not, defining the logic functions. The point is that by putting the appropriate pattern of transistors in the grids, you can create any function. For the division PLA, there are has 22 inputs (the 11 bits from the divisor and partial remainder indices, along with their complements) and two outputs, as shown below.13
A PLA is more compact than a ROM if the structure of the function allows it to be expressed with a small number of terms.14 One difficulty with a PLA is figuring out how to express the function with the minimum number of terms to make the PLA as small as possible. It turns out that this problem is NP-complete in general. Intel used a program called Espresso to generate compact PLAs using heuristics.15
The diagram below shows the division PLA in the Pentium. The PLA has 120 rows, split into two 60-row parts with support circuitry in the middle.16 The 11 table input bits go into the AND plane drivers in the middle, which produce the 22 inputs to the PLA (each table input and its complement). The outputs from the AND plane transistors go through output buffers and are fed into the OR plane. The outputs from the OR plane go through additional buffers and logic in the center, producing two output bits, indicating a ±1 or ±2 quotient. The image below shows the updated PLA that fixes the bug; the faulty PLA looks similar except the transistor pattern is different. In particular, the updated PLA has 46 unused rows at the bottom while the original, faulty PLA has 8 unused rows.
The image below shows part of the AND plane of the PLA. At each point in the grid, a transistor can be present or absent. The pattern of transistors in a row determines the logic term for that row. The vertical doped silicon lines (green) are connected to ground. The vertical polysilicon lines (red) are driven with the input bit pattern. If a polysilicon line crosses doped silicon, it forms a transistor (orange) that will pull that row to ground when activated.17 A metal line connects all the transistor rows in a row to produce the output; most of the metal has been removed, but some metal lines are visible at the right.
By carefully examining the PLA under a microscope, I extracted the pattern of transistors in the PLA grid. (This was somewhat tedious.) From the transistor pattern, I could determine the equations for each PLA row, and then generate the contents of the lookup table. Note that the transistors in the PLA don't directly map to the table contents (unlike a ROM). Thus, there is no specific place for transistors corresponding to the 5 missing table entries.
The left-hand side of the PLA implements the OR planes (below). The OR plane determines if the row output produces a quotient of 1 or 2. The OR plane is oriented 90° relative to the AND plane: the inputs are horizontal polysilicon lines (red) while the output lines are vertical. As before, a transistor (orange) is formed where polysilicon crosses doped silicon. Curiously, each OR plane has four outputs, even though the PLA itself has two outputs.18
Next, I'll show exactly how the AND plane produces a term. For the division table, the inputs are the 7 partial remainder bits and 4 divisor bits, as explained earlier. I'll call the partial remainder bits p6p5p4p3.p2p1p0 and the divisor bits 1.d3d2d1d0. These 11 bits and their complements are fed vertically into the PLA as shown at the top of the diagram below. These lines are polysilicon, so they will form transistor gates, turning on the corresponding transistor when activated. The arrows at the bottom point to nine transistors in the first row. (It's tricky to tell if the polysilicon line passes next to doped silicon or over the silicon, so the transistors aren't always obvious.) Looking at the transistors and their inputs shows that the first term in the PLA is generated by p0p1p2p3p4'p5p6d1d2.
The diagram below is a closeup of the lookup table, showing how this PLA row assigns the value 1 to four table cells (dark blue). You can think of each term of the PLA as pattern-matching to a binary pattern that can include "don't care" values. The first PLA term (above) matches the pattern P=110.1111, D=x11x, where the "don't care" x values can be either 0 or 1. Since one PLA row can implement multiple table cells, the PLA is more efficient than a ROM; the PLA uses 112 rows, while a ROM would require 2048 rows.
Geometrically, you can think of each PLA term (row) as covering a rectangle or rectangles in the table. However, the rectangle can't be arbitrary, but must be aligned on a bit boundary. Note that each "bump" in the table boundary (magenta) requires a separate rectangle and thus a separate PLA row. (This will be important later.)
One PLA row can generate a large rectangle, filling in many table cells at once, if the region happens to be aligned nicely. For instance, the third term in the PLA matches d=xxxx, p=11101xx. This single PLA row efficiently fills in 64 table cells as shown below, replacing the 64 rows that would be required in a ROM.
To summarize, the pattern of transistors in the PLA implements a set of equations, which define the contents of the table, setting the quotient to 1 or 2 as appropriate. Although the table has 2048 entries, the PLA represents the contents in just 112 rows. By carefully examining the transistor pattern, I determined the table contents in a faulty Pentium and a fixed Pentium.
As shown earlier, the lookup table has regions corresponding to quotient digits of +2, +1, 0, -1, and -2. These regions have irregular, slanted shapes, defined by mathematical bounds. In this section, I'll explain these mathematical bounds since they are critical to understanding how the Pentium bug occurred.
The essential step of the division algorithm is to divide the partial remainder p by the divisor d to get the quotient digit. The following diagram shows how p/d determines the quotient digit. The ratio p/d will define a point on the line at the top. (The point will be in the range [-8/3, 8/3] for mathematical reasons.) The point will fall into one of the five lines below, defining the quotient digit q. However, the five quotient regions overlap; if p/d is in one of the green segments, there are two possible quotient digits. The next part of the diagram illustrates how subtracting q*d from the partial remainder p shifts p/d into the middle, between -2/3 and 2/3. Finally, the result is multiplied by 4 (shifted left by two bits), expanding19 the interval back to [-8/3, 8/3], which is the same size as the original interval. The 8/3 bound may seem arbitrary, but the motivation is that it ensures that the new interval is the same size as the original interval, so the process can be repeated. (The bounds are all thirds for algebraic reasons; the value 3 comes from base 4 minus 1.20)
Note that the SRT algorithm has some redundancy, but cannot handle q values that are "too wrong". Specifically, if p/d is in a green region, then either of two q values can be selected. However, the algorithm cannot recover from a bad q value in general. The relevant case is that if q is supposed to be 2 but 0 is selected, the next partial remainder will be outside the interval and the algorithm can't recover. This is what causes the FDIV bug.
The diagram below shows the structure of the SRT lookup table (also called the P-D table since the axes are p and d). Each bound in the diagram above turns into a line in the table. For instance, the green segment above with p/d between 4/3 and 5/3 turns into a green region in the table below with 4/3 d ≤ p ≤ 5/3 d. These slanted lines show the regions in which a particular quotient digit q can be used.
The lookup table in the Pentium is based on the above table, quantized with a q value in each cell. However, there is one more constraint to discuss.
The Pentium's division circuitry uses a special circuit to perform addition and subtraction efficiently: the carry-save adder. One consequence of this adder is that each access to the lookup table may go to the cell just below the "right" cell. This is expected and should be fine, but in very rare and complicated circumstances, this behavior causes an access to one of the Pentium's five missing cells, triggering the division bug. In this section, I'll discuss why the division circuitry uses a carry-save adder, how the carry-save adder works, and how the carry-save adder triggers the FDIV bug.
The problem with addition is that carries make addition slow. Consider calculating 99999+1 by hand. You'll start with 9+1=10, then carry the one, generating another carry, which generates another carry, and so forth, until you go through all the digits. Computer addition has the same problem. If you're adding, say, two 64-bit numbers, the low-order bits can generate a carry that then propagates through all 64 bits. The time for the carry signal to go through 64 layers of circuitry is significant and can limit CPU performance. As a result, CPUs use special circuits to make addition faster.
The Pentium's division circuitry uses an unusual adder circuit called a carry-save adder to add (or subtract) the divisor and the partial remainder. A carry-save adder speeds up addition if you are performing a bunch of additions, as happens during division. The idea is that instead of adding a carry to each digit as it happens, you hold onto the carries in a separate word. As a decimal example, 499+222 would be 611 with carries 011; you don't carry the one to the second digit, but hold onto it. The next time you do an addition, you add in the carries you saved previously, and again save any new carries. The advantage of the carry-save adder is that the sum and carry at each digit position can be computed in parallel, which is fast. The disadvantage is that you need to do a slow addition at the end of the sequence of additions to add in the remaining carries to get the final answer. But if you're performing multiple additions (as for division), the carry-save adder is faster overall.
The carry-save adder creates a problem for the lookup table. We need to use the partial remainder as an index into the lookup table. But the carry-save adder splits the partial remainder into two parts: the sum bits and the carry bits. To get the table index, we need to add the sum bits and carry bits together. Since this addition needs to happen for every step of the division, it seems like we're back to using a slow adder and the carry-save adder has just made things worse.
The trick is that we only need 7 bits of the partial remainder for the table index, so we can use a different type of adder—a carry-lookahead adder—that calculates each carry in parallel using brute force logic. The logic in a carry-lookahead adder gets more and more complex for each bit so a carry-lookahead adder is impractical for large words, but it is practical for a 7-bit value.
The photo below shows the carry-lookahead adder used by the divider. Curiously, the adder is an 8-bit adder but only 7 bits are used; perhaps the 8-bit adder was a standard logic block at Intel.21 I'll just give a quick summary of the adder here, and leave the details for another post. At the top, logic gates compute signals in parallel for each of the 8 pairs of inputs: sum, carry generate, and carry propagate. Next, the complex carry-lookahead logic determines in parallel if there will be a carry at each position. Finally, XOR gates apply the carry to each bit. The circuitry in the middle is used for testing; see the footnote.22 At the bottom, the drivers amplify control signals for various parts of the adder and send the PLA output to other parts of the chip.23 By counting the blocks of repeated circuitry, you can see which blocks are 8 bits wide, 11, bits wide, and so forth. The carry-lookahead logic is different for each bit, so there is no repeated structure.
The carry-save and carry-lookahead adders may seem like implementation trivia, but they are a critical part of the FDIV bug because they change the constraints on the table. The cause is that the partial remainder is 64 bits,24 but the adder that computes the table index is 7 bits. Since the rest of the bits are truncated before the sum, the partial remainder sum for the table index can be slightly lower than the real partial remainder. Specifically, the table index can be one cell lower than the correct cell, an offset of 1/8. Recall the earlier diagram with diagonal lines separating the regions. Some (but not all) of these lines must be shifted down by 1/8 to account for the carry-save effect, but Intel made the wrong adjustment, which is the root cause of the FDIV error. (This effect was well-known at the time and mentioned in papers on SRT division, so Intel shouldn't have gotten it wrong.)
An interesting thing about the FDIV bug is how extremely rare it is. With 5 bad table entries out of 2048, you'd expect erroneous divides to be very common. However, for complicated mathematical reasons involving the carry-save adder the missing table entries are almost never encountered: only about 1 in 9 billion random divisions will encounter a problem. To hit a missing table entry, you need an "unlucky" result from the carry-save adder multiple times in a row, making the odds similar to winning the lottery, if the lottery prize were a division error.25
I consider the diagram below to be the "smoking gun" that explains how the FDIV bug happens: the top magenta line should be above the sloping black line, but it crosses the black line repeatedly. The magenta line carefully stays above the gray line, but that's the wrong line. In other words, Intel picked the wrong bounds line when defining the +2 region of the table. In this section, I'll explain why that causes the bug.
The diagram is colored according to the quotient values stored in the Pentium's lookup table: yellow is +2, blue is +1, and white is 0, with magenta lines showing the boundaries between different values. The diagonal black lines are the mathematical constraints on the table, defining the region that must be +2, the region that can be +1 or +2, the region that must be +1, and so forth. For the table to be correct, each cell value in the table must satisfy these constraints. The middle magenta line is valid: it remains between the two black lines (the redundant +1 or +2 region), so all the cells that need to be +1 are +1 and all the cells that need to be +2 are +2, as required. Likewise, the bottom magenta line remains between the black lines. However, the top magenta line is faulty: it must remain above the top black line, but it crosses the black line. The consequence is that some cells that need to be +2 end up holding 0: these are the missing cells that caused the FDIV bug.
Note that the top magenta line stays above the diagonal gray line while following it as closely as possible. If the gray line were the correct line, the table would be perfect. Unfortunately, Intel picked the wrong constraint line for the table's upper bound when the table was generated.26
But why are some diagonal lines lowered by 1/8 and other lines are not lowered? As explained in the previous section, as a consequence of the carry-save adder truncation, the table lookup may end up one cell lower than the actual p value would indicate, i.e. the p value for the table index is 1/8 lower than that actual value. Thus, both the correct cell and the cell below must satisfy the SRT constraints. Thus, the line moves down if that makes the constraints stricter but does not move down if that would expand the redundant area. In particular, the top line must not be move down, but clearly Intel moved the line down and generated the faulty lookup table.
Intel, however, has a different explanation for the bug. The Intel white paper states that the problem was in a script that downloaded the table into a PLA: an error caused the script to omit a few entries from the PLA.27 I don't believe this explanation: the missing terms match a mathematical error, not a copying error. I suspect that Intel's statement is technically true but misleading: they ran a C program (which they called a script) to generate the table but the program had a mathematical error in the bounds.
In his book "The Pentium Chronicles", Robert Colwell, architect of the Pentium Pro, provides a different explanation of the FDIV bug. Colwell claims that the Pentium design originally used the same lookup table as the 486, but shortly before release, the engineers were pressured by management to shrink the circuitry to save die space. The engineers optimized the table to make it smaller and had a proof that the optimization would work. Unfortunately, the proof was faulty, but the testers trusted the engineers and didn't test the modification thoroughly, causing the Pentium to be released with the bug. The problem with this explanation is that the Pentium was designed from the start with a completely different division algorithm from the 486: the Pentium uses radix-4 SRT, while the 486 uses standard binary division. Since the 486 doesn't have a lookup table, the story falls apart. Moreover, the PLA could trivially have been made smaller by removing the 8 unused rows, so the engineers clearly weren't trying to shrink it. My suspicion is that since Colwell developed the Pentium Pro in Oregon but the original Pentium was developed in California, Colwell didn't get firsthand information on the Pentium problems.
Intel's fix for the bug was straightforward but also surprising. You'd expect that Intel added the five missing table values to the PLA, and this is what was reported at the time. The New York Times wrote that Intel fixed the flaw by adding several dozen transistors to the chip. EE Times wrote that "The fix entailed adding terms, or additional gate-sequences, to the PLA."
However, the updated PLA (below) shows something entirely different. The updated PLA is exactly the same size as the original PLA. However, about 1/3 of the terms were removed from the PLA, eliminating hundreds of transistors. Only 74 of the PLA's 120 rows are used, and the rest are left empty. (The original PLA had 8 empty rows.) How could removing terms from the PLA fix the problem?
The explanation is that Intel didn't just fill in the five missing table entries with the correct value of 2. Instead, Intel filled all the unused table entries with 2, as shown below. This has two effects. First, it eliminates any possibility of hitting a mistakenly-empty entry. Second, it makes the PLA equations much simpler. You might think that more entries in the table would make the PLA larger, but the number of PLA terms depends on the structure of the data. By filling the unused cells with 2, the jagged borders between the unused regions (white) and the "2" regions (yellow) disappear. As explained earlier, a large rectangle can be covered by a single PLA term, but a jagged border requires a lot of terms. Thus, the updated PLA is about 1/3 smaller than the original, flawed PLA. One consequence is that the terms in the new PLA are completely different from the terms in the old PLA so one can't point to the specific transistors that fixed the bug.
The image below shows the first 14 rows of the faulty PLA and the first 14 rows of the fixed PLA. As you can see, the transistor pattern (and thus the PLA terms) are entirely different. The doped silicon is darkened in the second image due to differences in how I processed the dies to remove the metal layers.
How important is the Pentium bug? This became a highly controversial topic. A failure of a random division operation is very rare: about one in 9 billion values will trigger the bug. Moreover, an erroneous division is still mostly accurate: the error is usually in the 9th or 10th decimal digit, with rare worst-case error in the 4th significant digit. Intel's whitepaper claimed that a typical user would encounter a problem once every 27,000 years, insignificant compared to other sources of error such as DRAM bit flips. Intel said: "Our overall conclusion is that the flaw in the floating point unit of the Pentium processor is of no concern to the vast majority of users. A few users of applications in the scientific/engineering and financial engineering fields may need to employ either an updated processor without the flaw or a software workaround."
However, IBM performed their own analysis,29 suggesting that the problem could hit customers every few days, and IBM suspended Pentium sales. (Coincidentally, IBM had a competing processor, the PowerPC.) The battle made it to major newspapers; the Los Angeles Times split the difference with Study Finds Both IBM, Intel Off on Error Rate. Intel soon gave in and agreed to replace all the Pentiums, making the issue moot.
I mostly agree with Intel's analysis. It appears that only one person (Professor Nicely) noticed the bug in actual use.28 The IBM analysis seems contrived to hit numbers that trigger the error. Most people would never hit the bug and even if they hit it, a small degradation in floating-point accuracy is unlikely to matter to most people. Looking at society as a whole, replacing the Pentiums was a huge expense for minimal gain. On the other hand, it's reasonable for customers to expect an accurate processor.
Note that the Pentium bug is deterministic: if you use a specific divisor and dividend that trigger the problem, you will get the wrong answer 100% of the time. Pentium engineer Ken Shoemaker suggested that the outcry over the bug was because it was so easy for customers to reproduce. It was hard for Intel to argue that customers would never encounter the bug when customers could trivially see the bug on their own computer, even if the situation was artificial.
The FDIV bug is one of the most famous processor bugs. By examining the die, it is possible to see exactly where it is on the chip. But Intel has had other important bugs. Some early 386 processors had a 32-bit multiply problem. Unlike the deterministic FDIV bug, the 386 would unpredictably produce the wrong results under particular temperature/voltage/frequency conditions. The underlying issue was a layout problem that didn't provide enough electrical margin to handle the worst-case situation. Intel sold the faulty chips but restricted them to the 16-bit market; bad chips were labeled "16 BIT S/W ONLY", while the good processors were marked with a double sigma. Although Intel had to suffer through embarrassing headlines such as Some 386 Systems Won't Run 32-Bit Software, Intel Says, the bug was soon forgotten.
Another memorable Pentium issue was the "F00F bug", a problem where a particular instruction sequence starting with F0 0F would cause the processor to lock up until rebooted.30 The bug was found in 1997 and solved with an operating system update. The bug is presumably in the Pentium's voluminous microcode. The microcode is too complex for me to analyze, so don't expect a detailed blog post on this subject. :-)
You might wonder why Intel needed to release a new revision of the Pentium to fix the FDIV bug, rather than just updating the microcode. The problem was that microcode for the Pentium (and earlier processors) was hard-coded into a ROM and couldn't be modified. Intel added patchable microcode to the Pentium Pro (1995), allowing limited modifications to the microcode. Intel originally implemented this feature for chip debugging and testing. But after the FDIV bug, Intel realized that patchable microcode was valuable for bug fixes too.31 The Pentium Pro stores microcode in ROM, but it also has a static RAM that holds up to 60 microinstructions. During boot, the BIOS can load a microcode patch into this RAM. In modern Intel processors, microcode patches have been used for problems ranging from the Spectre vulnerability to voltage problems.
As the number of transistors in a processor increased exponentially, as described by Moore's Law, processors used more complex circuits and algorithms. Division is one example. Early microprocessors such as the Intel 8080 (1974, 6000 transistors) had no hardware support for division or floating point arithmetic. The Intel 8086 (1978, 29,000 transistors) implemented integer division in microcode but required the 8087 coprocessor chip for floating point. The Intel 486 (1989, 1.2 million transistors) added floating-point support on the chip. The Pentium (1993, 3.1 million transistors) moved to the faster but more complicated SRT division algorithm. The Pentium's division PLA alone has roughly 4900 transistor sites, more than a MOS Technology 6502 processor—one component of the Pentium's division circuitry uses more transistors than an entire 1975 processor.
The long-term effect of the FDIV bug on Intel is a subject of debate. On the one hand, competitors such as AMD benefitted from Intel's error. AMD's ads poked fun at the Pentium's problems by listing features of AMD's chips such as "You don't have to double check your math" and "Can actually handle the rigors of complex calculations like division." On the other hand, Robert Colwell, architect of the Pentium Pro, said that the FDIV bug may have been a net benefit to Intel as it created enormous name recognition for the Pentium, along with a demonstration that Intel was willing to back up its brand name. Industry writers agreed; see The Upside of the Pentium Bug. In any case, Intel survived the FDIV bug; time will tell how Intel survives its current problems.
I plan to write more about the implementation of the Pentium's PLA, the adder, and the test circuitry. Until then, you may enjoy reading about the Pentium Navajo rug. (The rug represents the P54C variant of the Pentium, so it is safe from the FDIV bug.) Thanks to Bob Colwell and Ken Shoemaker for helpful discussions.
The book Inside Intel says that Vin Dham, the "Pentium czar", found the FDIV problem in May 1994. The book "The Pentium Chronicles" says that Patrice Roussel, the floating-point architect for Intel's upcoming Pentium Pro processor, found the FDIV problem in Summer 1994. I suspect that the bug was kept quiet inside Intel and was discovered more than once. ↩
The divisor being a prime number has nothing to do with the bug. It's just a coincidence that the problem was found during research with prime numbers. ↩
See Nicely's FDIV page for more information on the bug and its history. Other sources are the books Creating the Digital Future, The Pentium Chronicles, and Inside Intel. The New York Times wrote about the bug: Flaw Undermines Accuracy of Pentium Chips. Computerworld wrote Intel Policy Incites User Threats on threats of a class-action lawsuit. IBM's response is described in IBM Deals Blow to a Rival as it Suspends Pentium Sales ↩
Talk show host David Letterman joked about the Pentium on December 15: "You know what goes great with those defective Pentium chips? Defective Pentium salsa!" Although a list of Letterman-style top ten Pentium slogans circulated, the list was a Usenet creation. There's a claim that Jay Leno also joked about the Pentium, but I haven't found verification. ↩
Processors have many more bugs than you might expect. Intel's 1995 errata list for the Pentium had "21 errata (including the FDIV problem), 4 changes, 16 clarifications, and 2 documentation changes." See Pentium Processor Specification Update and Intel Releases Pentium Errata List. ↩
Intel published full-page newspaper ads apologizing for its handling of the problem, stating: "What Intel continues to believe is an extremely minor technical problem has taken on a life of its own."
Tim Coe's reverse engineering of the Pentium divider was described on the Usenet group comp.sys.intel, archived here. To summarize, Andreas Kaiser found 23 failing reciprocals. Tim Coe determined that most of these failing reciprocals were of the form 3*(2^(K+30)) - 1149*(2^(K-(2*J))) - delta*(2^(K-(2*J))). He recognized that the factor of 2 indicated a radix-4 divider. The extremely low probability of error indicated the presence of a carry save adder; the odds of both the sum and carry bits getting long patterns of ones were very low. Coe constructed a simulation of the divider that matched the Pentium's behavior and noted which table entries must be faulty. ↩
The main papers on the FDIV bug are Computational Aspects of the Pentium Affair, It Takes Six Ones to Reach a Flaw, The Mathematics of the Pentium Division Bug, The Truth Behind the Pentium Bug, Anatomy of the Pentium Bug, and Risk Analysis of the Pentium Bug. Intel's whitepaper is Statistical Analysis of Floating Point Flaw in the Pentium Processor; I archived IBM's study here. ↩
The Pentium uses floating point numbers that follow the IEEE 754 standard. Internally, floating point numbers are represented with 80 bits: 1 bit for the sign, 15 bits for the exponent, and 64 bits for the significand. Externally, floating point numbers are 32-bit single-precision numbers or 64-bit double-precision numbers. Note that the number of significand bits limits the accuracy of a floating-point number. ↩
The SRT division algorithm is named after the three people who independently created it in 1957-1958: Sweeney at IBM, Robertson at the University of Illinois, and Tocher at Imperial College London. The SRT algorithm was developed further by Atkins in his PhD research (1970).
The SRT algorithm became more practical in the 1980s as chips became denser. Taylor implemented the SRT algorithm on a board with 150 chips in 1981. The IEEE floating point standard (1985) led to a market for faster floating point circuitry. For instance, the Weitek 4167 floating-point coprocessor chip (1989) was designed for use with the Intel 486 CPU (datasheet) and described in an influential paper. Another important SRT implementation is the MIPS R3010 (1988), the coprocessor for the R3000 RISC processor. The MIPS R3010 uses radix-4 SRT for division with 9 bits from the partial remainder and 9 bits from the divisor, making for a larger lookup table and adder than the Pentium (link).
To summarize, when Intel wanted to make division faster on the Pentium (1993), the SRT algorithm was a reasonable choice. Competitors had already implemented SRT and multiple papers explained how SRT worked. The implementation should have been straightforward and bug-free. ↩
The dimensions of the lookup table can't be selected arbitrarily. In particular, if the table is too small, a cell may need to hold two different q values, which isn't possible. Note that constructing the table is only possible due to the redundancy of SRT. For instance, if some values in the call require q=1 and other values require q=1 or 2, then the value q=1 can be assigned to the cell. ↩
In the white paper, Intel calls the PLA a Programmable Lookup Array, but that's an error; it's a Programmable Logic Array. ↩
I'll explain a PLA in a bit more detail in this footnote. An example of a sum-of-products formula with inputs a and b is ab' + a'b + ab. This formula has three sum terms, so it requires three rows in the PLA. However, this formula can be reduced to a + b, which uses a smaller two-row PLA. Note that any formula can be trivially expressed with a separate product term for each 1 output in the truth table. The hard part is optimizing the PLA to use fewer terms. The original PLA patent is probably MOS Transistor Integrated Matrix from 1969. ↩
A ROM and a PLA have many similarities. You can implement a ROM with a PLA by using the AND terms to decode addresses and the OR terms to hold the data. Alternatively, you can replace a PLA with a ROM by putting the function's truth table into the ROM. ROMs are better if you want to hold arbitrary data that doesn't have much structure (such as the microcode ROMs). PLAs are better if the functions have a lot of underlying structure. The key theoretical difference between a ROM and a PLA is that a ROM activates exactly one row at a time, corresponding to the address, while a PLA may activate one row, no rows, or multiple rows at a time. Another alternative for representing functions is to use logic gates directly (known as random logic); moving from the 286 to the 386, Intel replaced many small PLAs with logic gates, enabled by improvements in the standard-cell software. Intel's design process is described in Coping with the Complexity of Microprocessor Design. ↩
In 1982, Intel developed a program called LOGMIN to automate PLA design. The original LOGMIN used an exhaustive exponential search, limiting its usability. See A Logic Minimizer for VLSI PLA Design. For the 386, Intel used Espresso, a heuristic PLA minimizer that originated at IBM and was developed at UC Berkeley. Intel probably used Espresso for the Pentium, but I can't confirm that. ↩
The Pentium's PLA is split into a top half and a bottom half, so you might expect the top half would generate a quotient of 1 and the bottom half would generate a quotient of 2. However, the rows for the two quotients are shuffled together with no apparent pattern. I suspect that the PLA minimization software generated the order arbitrarily. ↩
Conceptually, the PLA consists of AND gates feeding into OR gates. To simplify the implementation, both layers of gates are actually NOR gates. Specifically, if any transistor in a row turns on, the row will be pulled to ground, producing a zero. De Morgan's laws show that the two approaches are the same, if you invert the inputs and outputs. I'm ignoring this inversion in the diagrams.
Note that each square can form a transistor on the left, the right, or both. The image must be examined closely to distinguish these cases. Specifically, if the polysilicon line produces a transistor, horizontal lines are visible in the polysilicon. If there are no horizontal lines, the polysilicon passes by without creating a transistor. ↩
Each OR plane has four outputs, so there are eight outputs in total. These outputs are combined with logic gates to generate the desired two outputs (quotient of 1 or 2). I'm not sure why the PLA is implemented in this fashion. Each row alternates between an output on the left and an output on the right, but I don't think this makes the layout any denser. As far as I can tell, the extra outputs just waste space. One could imagine combining the outputs in a clever way to reduce the number of terms, but instead the outputs are simply OR'd together. ↩
The dynamics of the division algorithm are interesting. The computation of a particular division will result in the partial remainder bouncing from table cell to table cell, while remaining in one column of the table. I expect this could be analyzed in terms of chaotic dynamics. Specifically, the partial remainder interval is squished down by the subtraction and then expanded when multiplied by 4. This causes low-order bits to percolate upward so the result is exponentially sensitive to initial conditions. I think that the division behavior satisfies the definition of chaos in Dynamics of Simple Maps, but I haven't investigated this in detail.
You can see this chaotic behavior with a base-10 division, e.g.
compare 1/3.0001 to 1/3.0002:
1/3.0001=0.333322222592580247325089163694543515216...
1/3.0002=0.333311112592493833744417038864075728284...
Note that the results start off the same but are completely divergent by 15 digits.
(The division result itself isn't chaotic, but the sequence of digits is.)
I tried to make a fractal out of the SRT algorithm and came up with the image below. There are 5 bands for convergence, each made up of 5 sub-bands, each made up of 5 sub-sub bands, and so on, corresponding to the 5 q values.
The algebra behind the bound of 8/3 is that p (the partial remainder) needs to be in an interval that stays the same size each step. Each step of division computes pnew = (pold - q*d)*4. Thus, at the boundary, with q=2, you have p = (p-2*d)*4, so 3p=8d and thus p/d = 8/3. Similarly, the other boundary, with q=-2, gives you p/d = -8/3. ↩
I'm not completely happy with the 8-bit carry-lookahead adder. Coe's mathematical analysis in 1994 showed that the carry-lookahead adder operates on 7 bits. The adder in the Pentium has two 8-bit inputs connected to another part of the division circuit. However, the adder's bottom output bit is not connected to anything. That would suggest that the adder is adding 8 bits and then truncating to 7 bits, which would reduce the truncation error compared to a 7-bit adder. However, when I simulate the division algorithm this way, the FDIV bug doesn't occur. Wiring the bottom input bits to 0 would explain the behavior, but that seems pointless. I haven't examined the circuitry that feeds the adder, so I don't have a conclusive answer. ↩
Half of the circuitry in the adder block is used to test the lookup table. The reason is that a chip such as the Pentium is very difficult to test: if one out of 3.1 million transistors goes bad, how do you detect it? For a simple processor like the 8080, you can run through the instruction set and be fairly confident that any problem would turn up. But with a complex chip, it is almost impossible to come up with an instruction sequence that would test every bit of the microcode ROM, every bit of the cache, and so forth. Starting with the 386, Intel added circuitry to the processor solely to make testing easier; about 2.7% of the transistors in the 386 were for testing.
To test a ROM inside the processor, Intel added circuitry to scan the entire ROM and checksum its contents. Specifically, a pseudo-random number generator runs through each address, while another circuit computes a checksum of the ROM output, forming a "signature" word. At the end, if the signature word has the right value, the ROM is almost certainly correct. But if there is even a single bit error, the checksum will be wrong and the chip will be rejected. The pseudo-random numbers and the checksum are both implemented with linear feedback shift registers (LFSR), a shift register along with a few XOR gates to feed the output back to the input. For more information on testing circuitry in the 386, see Design and Test of the 80386, written by Pat Gelsinger, who became Intel's CEO years later. Even with the test circuitry, 48% of the transistor sites in the 386 were untested. The instruction-level test suite to test the remaining circuitry took almost 800,000 clock cycles to run. The overhead of the test circuitry was about 10% more transistors in the blocks that were tested.
In the Pentium, the circuitry to test the lookup table PLA is just below the 7-bit adder. An 11-bit LFSR creates the 11-bit input value to the lookup table. A 13-bit LFSR hashes the two-bit quotient result from the PLA, forming a 13-bit checksum. The checksum is fed serially to test circuitry elsewhere in the chip, where it is merged with other test data and written to a register. If the register is 0 at the end, all the tests pass. In particular, if the checksum is correct, you can be 99.99% sure that the lookup table is operating as expected. The ironic thing is that this test circuit was useless for the FDIV bug: it ensured that the lookup table held the intended values, but the intended values were wrong.
Why did Intel generate test addresses with a pseudo-random sequence instead of a sequential counter? It turns out that a linear feedback shift register (LFSR) is slightly more compact than a counter. This LFSR trick was also used in a touch-tone chip and the program counter of the Texas Instruments TMS 1000 microcontroller (1974). In the TMS 1000, the program counter steps through the program pseudo-randomly rather than sequentially. The program is shuffled appropriately in the ROM to counteract the sequence, so the program executes as expected and a few transistors are saved. ↩
One unusual feature of the Pentium is that it uses BiCMOS technology: both bipolar and CMOS transistors. Note the distinctive square boxes in the driver circuitry; these are bipolar transistors, part of the high-speed drivers.
I think the partial remainder is actually 67 bits because there are three extra bits to handle rounding. Different parts of the floating-point datapath have different widths, depending on what width is needed at that point. ↩
In this long footnote, I'll attempt to explain why the FDIV bug is so rare, using heatmaps. My analysis of Intel's lookup table shows several curious factors that almost cancel out, making failures rare but not impossible. (For a rigorous explanation, see It Takes Six Ones to Reach a Flaw and The Mathematics of the Pentium Division Bug. These papers explain that, among other factors, a bad divisor must have six consecutive ones in positions 5 through 10 and the division process must go through nine specific steps, making a bad result extremely uncommon.)
The diagram below shows a heatmap of how often each table cell is accessed when simulating a generic SRT algorithm with a carry-save adder. The black lines show the boundaries of the quotient regions in the Pentium's lookup table. The key point is that the top colored cell in each column is above the black line, so some table cells are accessed but are not defined in the Pentium. This shows that the Pentium is missing 16 entries, not just the 5 entries that are usually discussed. (For this simulation, I generated the quotient digit directly from the SRT bounds, rather than the lookup table, selecting the digit randomly in the redundant regions.)
The diagram is colored with a logarithmic color scale. The blue cells are accessed approximately uniformly. The green cells at the boundaries are accessed about 2 orders of magnitude less often. The yellow-green cells are accessed about 3 orders of magnitude less often. The point is that it is hard to get to the edge cells since you need to start in the right spot and get the right quotient digit, but it's not extraordinarily hard.
(The diagram also shows an interesting but ultimately unimportant feature of the Pentium table: at the bottom of the diagram, five white cells are above the back line. This shows that the Pentium assigns values to five table cells that can't be accessed. (This was also mentioned in "The Mathematics of the Pentium Bug".) These cells are in the same columns as the 5 missing cells, so it would be interesting if they were related to the missing cells. But as far as I can tell, the extra cells are due to using a bound of "greater or equals" rather than "greater", unrelated to the missing cells. In any case, the extra cells are harmless.)
The puzzling factor is that if the Pentium table has 16 missing table cells, and the SRT uses these cells fairly often, you'd expect maybe 1 division out of 1000 or so to be wrong. So why are division errors extremely rare?
It turns out that the structure of the Pentium lookup table makes some table cells inaccessible. Specifically, the table is arbitrarily biased to pick the higher quotient digit rather than the lower quotient digit in the redundant regions. This has the effect of subtracting more from the partial remainder, pulling the partial remainder away from the table edges. The diagram below shows a simulation using the Pentium's lookup table and no carry-save adder. Notice that many cells inside the black lines are white, indicating that they are never accessed. This is by coincidence, due to arbitrary decisions when constructing in the lookup table. Importantly, the missing cells just above the black line are never accessed, so the missing cells shouldn't cause a bug.
Thus, Intel almost got away with the missing table entries. Unfortunately, the carry-save adder makes it possible to reach some of the otherwise inaccessible cells. Because the output from the carry-save adder is truncated, the algorithm can access the table cell below the "right" cell. In the redundant regions, this can yield a different (but still valid) quotient digit, causing the next partial remainder to end up in a different cell than usual. The heatmap below shows the results.
In particular, five cells above the black line can be reached: these are instances of the FDIV bug. These cells are orange, indicating that they are about 9 orders of magnitude less likely than the rest of the cells. It's almost impossible to reach these cells, requiring multiple "unlucky" values in a row from the carry-save adder. To summarize, the Pentium lookup table has 16 missing cells. Purely by coincidence, the choices in the lookup table make many cells inaccessible, which almost counteracts the problem. However, the carry-save adder provides a one-in-a-billion path to five of the missing cells, triggering the FDIV bug.
One irony is that if division errors were more frequent, Intel would have caught the FDIV bug before shipping. But if division errors were substantially less frequent, no customers would have noticed the bug. Inconveniently, the frequency of errors fell into the intermediate zone: errors were too rare for Intel to spot them, but frequent enough for a single user to spot them. (This makes me wonder what other astronomically infrequent errors may be lurking in processors.) ↩
Anatomy of the Pentium Bug reached a similar conclusion, stating "The [Intel] White Paper attributes the error to a script that incorrectly copied values; one is nevertheless tempted to wonder whether the rule for lowering thresholds was applied to the 8D/3 boundary, which would be an incorrect application because that boundary is serving to bound a threshold from below." (That paper also hypothesizes that the table was compressed to 6 columns, a hypothesis that my examination of the die disproves.) ↩
The Intel white paper describes the underlying cause of the bug: "After the quantized P-D plot (lookup table) was numerically generated as in Figure 4-1, a script was written to download the entries into a hardware PLA (Programmable Lookup Array). An error was made in this script that resulted in a few lookup entries (belonging to the positive plane of the P-D plot) being omitted from the PLA." The script explanation is repeated in The Truth Behind the Pentium Bug: "An engineer prepared the lookup table on a computer and wrote a script in C to download it into a PLA (programmable logic array) for inclusion in the Pentium's FPU. Unfortunately, due to an error in the script, five of the 1066 table entries were not downloaded. To compound this mistake, nobody checked the PLA to verify the table was copied correctly." My analysis suggests that the table was copied correctly; the problem was that the table was mathematically wrong. ↩
It's not hard to find claims of people encountering the Pentium division bug, but these seem to be in the "urban legend" category. Either the problem is described second-hand, or the problem is unrelated to division, or the problem happened much too frequently to be the FDIV bug. It has been said that the game Quake would occasionally show the wrong part of a level due to the FDIV bug, but I find that implausible. The "Intel Inside—Don't Divide" Chipwreck describes how the division bug was blamed for everything from database and application server crashes to gibberish text. ↩
IBM's analysis of the error rate seems contrived, coming up with reasons to use numbers that are likely to cause errors. In particular, IBM focuses on slightly truncated numbers, either numbers with two decimal digits or hardcoded constants. Note that a slightly truncated number is much more likely to hit a problem because its binary representation will have multiple 1's in a row, a necessity to trigger the bug. Another paper Risk Analysis of the Pentium Bug claims a risk of one in every 200 divisions. It depends on "bruised integers", such as 4.999999, which are similarly contrived. I'll also point out that if you start with numbers that are "bruised" or otherwise corrupted, you obviously don't care about floating-point accuracy and shouldn't complain if the Pentium adds slightly more inaccuracy.
The book "Inside Intel" says that "the IBM analysis was quite wrong" and "IBM's intervention in the Pentium affair was not an example of the company on its finest behavior" (page 364). ↩
The F00F bug happens when an invalid compare-and-exchange instruction leaves the bus locked. The instruction is supposed to exchange with a memory location, but the invalid instruction specifies a register instead causing unexpected behavior. This is very similar to some undocumented instructions in the 8086 processor where a register is specified when memory is required; see my article Undocumented 8086 instructions, explained by the microcode. ↩
For details on the Pentium Pro's patchable microcode, see P6 Microcode Can Be Patched. But patchable microcode dates back much earlier. The IBM System/360 mainframes (1964) had microcode that could be updated in the field, either to fix bugs or to implement new features. These systems stored microcode on metalized Mylar sheets that could be replaced as necessary. In that era, semiconductor ROMs didn't exist, so Mylar sheets were also a cost-effective way to implement read-only storage. See TROS: How IBM mainframes stored microcode in transformers. ↩
I was studying the silicon die of the Pentium processor and noticed some puzzling structures where signal lines were connected to the silicon substrate for no apparent reason. Two examples are in the photo below, where the metal wiring (orange) connects to small square regions of doped silicon (gray), isolated from the rest of the circuitry. I did some investigation and learned that these structures are "antenna diodes," special diodes that protect the circuitry from damage during manufacturing. In this blog post, I discuss the construction of the Pentium and explain how these antenna diodes work.
Intel released the Pentium processor in 1993, starting a long-running brand of high-performance processors: the Pentium Pro, Pentium II, and so on. In this post, I'm studying the original Pentium, which has 3.1 million transistors.1 The die photo below shows the Pentium's fingernail-sized silicon die under a microscope. The chip has three layers of metal wiring on top of the silicon so the underlying silicon is almost entirely obscured.
Modern processors are built from CMOS circuitry, which uses two types of transistors: NMOS and PMOS. The diagram below shows how an NMOS transistor is constructed. A transistor can be considered a switch between the source and drain, controlled by the gate. The source and drain regions (green) consist of silicon doped with impurities to change its semiconductor properties, forming N+ silicon. The gate consists of a layer of polysilicon (red), separated from the silicon by an absurdly thin insulating oxide layer. Since the oxide layer is just a few hundred atoms thick,2 it is very fragile and easily damaged by excess voltage. (This is why CMOS chips are sensitive to static electricity.) As we will see, the oxide layer can also be damaged by voltage during manufacturing.
The Pentium processor is constructed from multiple layers. Starting at the bottom, the Pentium has millions of transistors similar to the diagram above. Polysilicon wiring on top of the silicon not only forms the transistor gates but also provides short-range wiring. Above that, three layers of metal wiring connect the parts of the chip. Roughly speaking, the bottom layer of metal connects to the silicon and polysilicon to construct logic gates from the transistors, while the upper layers of wiring travel longer distances, with one layer for signals traveling horizontally and the other layer for signals traveling vertically. Tiny tungsten plugs called vias provide connections between the different layers of wiring. A key challenge of chip design is routing, directing signals through the multiple layers of wiring while packing the circuitry as densely as possible.
The photo below shows a small region of the Pentium die with the three metal layers visible. The golden vertical lines are the top metal layer, formed from aluminum and copper. Underneath, you can see the horizontal wiring of the middle metal layer. The more complex wiring of the bottom metal layer can be seen, along with the silicon and polysilicon that form transistors. The small black dots are the tungsten vias that connect metal layers, while the larger dark circles are contacts with the underlying silicon or polysilicon. Near the bottom of the photo, the vertical gray bands are polysilicon lines, forming transistor gates. Although the chip appears flat, it has a three-dimensional structure with multiple layers of metal separated by insulating layers of silicon dioxide. This three-dimensional structure will be important in the discussion below. (The metal wiring is much denser over most of the chip; this region is one of the rare spots where all the layers are visible.)
The manufacturing process for an integrated circuit is extraordinarily complicated but I'll skip over most of the details and focus on how each metal layer is constructed, layer by layer. First, a uniform metal layer is constructed over the silicon wafer. Next, the desired pattern is produced on the surface using a process called photolithography: a light-sensitive chemical called "resist" is applied to the wafer and exposed to light through a patterned mask. The light hardens the resist, creating a protective coating with the pattern of the desired wiring. Finally, the unprotected metal is etched away, leaving the wiring.
In the early days of integrated circuits, the metal was removed with liquid acids, a process called wet etching. Inconveniently, wet etching tended to eat away metal underneath the edges of the mask, which became a problem as integrated circuits became denser and the wires needed to be thinner. The solution was dry etch, using a plasma to remove the metal. By applying a large voltage to plates above and below the chip, a gas such as HCl is ionized into a highly reactive plasma. This plasma attacks the surface (unless it is protected by the resist), removing the unwanted metal. The advantage of dry etching is that it can act vertically (anisotropically), providing more control over the line width.
Although plasma etching improved the etching process, it caused another problem: plasma-induced oxide damage, also called (metaphorically) the "antenna effect."3 The problem is that long metal wires on the chip could pick up an electrical charge from the plasma, producing a large voltage. As described earlier, the thin oxide layer under a transistor's gate is sensitive to voltage damage. The voltage induced by the plasma can destroy the transistor by blowing a hole through the gate oxide or it can degrade the transistor's performance by embedding charges inside the oxide layer.4
Several factors affect the risk of damage from the antenna effect. First, only the transistor's gate is sensitive to the induced voltage, due to the oxide layer. If the wire is also connected to a transistor's source or drain, the wire is "safe" since the source and drain provide connections to the chip's substrate, allowing the charge to dissipate harmlessly. Note that when the chip is completed, every transistor gate is connected to another transistor's source or drain (which provides the signal to the gate), so there is no risk of damage. Thus, the problem can only occur during manufacturing, with a metal line that is connected to a gate on one end but isn't connected on the other end. Moreover, the highest layer of metal is "safe" since everything is connected at that point. Another factor is that the induced voltage is proportional to the length of the metal wire, so short wires don't pose a risk. Finally, only the metal layer currently being etched poses a risk; since the lower layers are insulated by the thick oxide between layers, they won't pick up charge.
These factors motivate several ways to prevent antenna problems.5 First, a long wire can be broken into shorter segments, connected by jumpers on a higher layer. Second, moving long wires to the top metal layer eliminates problems.6 Third, diodes can be added to drain the charge from the wire; these are called "antenna diodes". When the chip is in use, the antenna diodes are reverse-biased so they have no electrical effect. But during manufacturing, the antenna diodes let charge flow to the substrate before it causes problems.
The third solution, the antenna diodes, explains the mysterious connections that I saw in the Pentium. In the diagram below, these diodes are visible on the die as square regions of doped silicon. The larger regions of doped silicon form PMOS transistors (upper) and NMOS transistors (lower). The polysilicon lines are faintly visible; they form transistor gates where they cross the doped silicon. (For this photo, I removed all the metal wiring.)
Confusingly, the antenna diodes look almost identical to "well taps", connections from the substrate to the chip's positive voltage supply, but have a completely different purpose. In the Pentium, the PMOS transistors are constructed in "wells" of N-type silicon. These wells must be raised to the chip's positive voltage, so there are numerous well tap connections from the positive supply to the wells. The well taps consist of squares of N+ doped silicon in the the N-type silicon well, providing an electrical connection. On the other hand, the antenna diodes also consist of N+ doped silicon, but embedded in P-type silicon. This forms a P-N junction that creates the diode.
In the Pentium, antenna diodes are used for only a small fraction of the wiring. The diodes require extra area on the die, so they are used only when necessary. Most of the antenna problems on the Pentium were apparently resolved through routing. Although the antenna diodes are relatively rare, they are still frequent enough that they caught my attention.
Antenna effects are still an issue in modern integrated circuits. Integrated circuit fabricators provide rules on the maximum allowable size of antenna wires for a particular manufacturing process.7 Software checks the design to ensure that the antenna rules are not violated, modifying the routing and inserting diodes as necessary. Violating the antenna rules can result in damaged chips and a very low yield, so it's more than just a theoretical issue.
Thanks to /r/chipdesign and Discord for discussion. If you're interested in the Pentium, I've written about standard cells in the Pentium, and the Pentium as a Navajo rug. Follow me on Mastodon (@kenshirriff@oldbytes.space) or Bluesky (@righto.com) or RSS for updates.
In this post, I'm looking at the Pentium model 80501 (codenamed P5). This model was soon replaced with a faster, lower-power version called the 80502 (P54C). Both are considered original Pentiums. ↩
IC manufacturing drives CPU performance states that gate oxide thickness was 100 to 300 angstroms in 1993. ↩
The wires are acting metaphorically as antennas, not literally, as they collect charge, not picking up radio waves.
Plasma-induced oxide damage gave rise to research and conferences in the 1990s to address this problem. The International Symposium on Plasma- and Process-Induced Damage started in 1996 and continued until 2003. Numerous researchers from semiconductor companies and academia studied the causes and effects of plasma damage. ↩
The damage is caused by "Fowler-Nordheim tunneling", where electrons tunnel through the oxide and cause damage. Flash memory uses this tunneling to erase the memory; the cumulative damage is why flash memory can only be written a limited number of times. ↩
Some relevant papers: Magnetron etching of polysilicon: Electrical damage (1991), Thin-oxide damage from gate charging during plasma processing (1992), Antenna protection strategy for ultra-thin gate MOSFETs (1998), Fixing antenna problem by dynamic diode dropping and jumper insertion (2000). The Pentium uses the "dynamic diode dropping" approach, adding antenna diodes only as needed, rather than putting them in every circuit. I noticed that the Pentium uses extension wires to put the diode in a more distant site if there is no room for the diode under the existing wiring. As an aside, the third paper uses the curious length unit of kµm; by calling 1000 µm a kµm, you can think in micrometers, even though this unit is normally called a mm. ↩
Sources say that routing signals on the top metal prevents antenna violations. However, I see several antenna diodes in the Pentium that are connected directly from the bottom metal (M1) through M2 to long lines on M3. These diodes seem redundant since the source/drain connections are in place by that time. So there are still a few mysteries... ↩
Foundries have antenna rules provided as part of the Process Design Kit (PDK). Here are the rules for MOSIS and SkyWater. I've focused on antenna effects from the metal wiring, but polysilicon and vias can also cause antenna damage. Thus, there are rules for these layers too. Polysilicon wiring is less likely to cause antenna problems, though, as it is usually restricted to short distances due to its higher resistance. ↩
Forbes recently published the Forbes 400 List for 2024, listing the 400 richest people in the United States. This inspired me to make a histogram to show the distribution of wealth in the United States. It turns out that if you put Elon Musk on the graph, almost the entire US population is crammed into a vertical bar, one pixel wide. Each pixel is $500 million wide, illustrating that $500 million essentially rounds to zero from the perspective of the wealthiest Americans.
The histogram above shows the wealth distribution in red. Note that the visible red line is one pixel wide at the left and disappears everywhere else—this is the important point: essentially the entire US population is in that first bar. The graph is drawn with the scale of 1 pixel = $500 million in the X axis, and 1 pixel = 1 million people in the Y axis. Away from the origin, the red line is invisible—a tiny fraction of a pixel tall since so few people have more than 500 million dollars.
Since the median US household wealth is about $190,000, half the population would be crammed into a microscopic red line 1/2500 of a pixel wide using the scale above. (The line would be much narrower than the wavelength of light so it would be literally invisible). The very rich are so rich that you could take someone with a thousand times the median amount of money, and they would still have almost nothing compared to the richest Americans. If you increased their money by a factor of a thousand yet again, you'd be at Bezos' level, but still well short of Elon Musk.
Another way to visualize the extreme distribution of wealth in the US is to imagine everyone in the US standing up while someone counts off millions of dollars, once per second. When your net worth is reached, you sit down. At the first count of $1 million, most people sit down, with 22 million people left standing. As the count continues—$2 million, $3 million, $4 million—more people sit down. After 6 seconds, everyone except the "1%" has taken their seat. As the counting approaches the 17-minute mark, only billionaires are left standing, but there are still days of counting ahead. Bill Gates sits down after a bit over one day, leaving 8 people, but the process is nowhere near the end. After about two days and 20 hours of counting, Elon Musk finally sits down.
The main source of data is the Forbes 400 List for 2024. Forbes claims there are 813 billionaires in the US here. Median wealth data is from the Federal Reserve; note that it is from 2022 and household rather than personal. The current US population estimate is from Worldometer. I estimated wealth above $500 million, extrapolating from 2019 data.
I made a similar graph in 2013; you can see my post here for comparison.
Disclaimers: Wealth data has a lot of sources of error including people vs households, what gets counted, and changing time periods, but I've tried to make this graph as accurate as possible. I'm not making any prescriptive judgements here, just presenting the data. Obviously, if you want to see the details of the curve, a logarithmic scale makes more sense, but I want to show the "true" shape of the curve. I should also mention that wealth and income are very different things; this post looks strictly at wealth.
One Netbook has been selling external graphics docks for handhelds, laptops and mini PCs for a few years. The original ONXEGPU launched in 2023 and featured an AMD Radeon RX 7600M XT GPU. A year later the company launched a ONEXGPU 2 with Radeon RX 7800M. Now the company is showing off a new model […]
The post ONEXGPU 3 graphics dock with Radeon RX 9070 XT GPU supports OCuLink and USB4 connections appeared first on Liliputing.
Intel’s Alder Lake-N and Twin Lake chips are cheap, low-power chips that have been popular options for entry-level mini PCs, laptops, and other devices like tablets and single-board PCs. But while these chips offer much better CPU and graphics performance than the Intel Celeron and Atom-branded chips they replaced, they’re based on architecture that’s several years […]
The post Intel Wildcat Lake 6-core chips for budget devices will replace Alder Lake-N and Twin Lake (eventually) appeared first on Liliputing.
The Asus ExpertCenter PN55 is a mini PC powered by an AMD Ryzen AI 400 “Gorgon Point” processor. Measuring 130 x 130 x 34mm (5.1″ x 5.1″ x 1.3″), it’s the same size and shape as last year’s PN54 and PN54-S1 systems. But top-tier configurations should offer a nice performance boost. That’s not because AMD’s Ryzen AI […]
The post Asus ExpertCenter PN55 is a mini PC with up to Ryzen AI 9 HX 470 appeared first on Liliputing.
The AYANEO AM03 is a small desktop computer with a design inspired by classic game consoles, although unlike some of the company’s other models, the AM03 doesn’t look like a clone of 1980s-era PCs or consoles so much as it takes some of it design cues from that era. Available in a black or cyan colors, […]
The post AYANEO launches AM03 mini PC with retro console-like design for $399 and up appeared first on Liliputing.
The Lenovo Yoga Mini i is a compact desktop computer with an aluminum chassis that measures about 130 x 130 x 48.6mm (5.12″ x 5.12″ x 1.91″). With support for up to an Intel Core Ultra X7 358H processor, it’s a reasonably powerful little computer. But what really makes it stand out is the design. Instead […]
The post Lenovo Yoga Mini i is a hockey puck-shaped mini PC with Intel Panther Lake inside appeared first on Liliputing.
Last year Lenovo launched the first laptop with a rollable display that could extend vertically to give you more screen space with the press of the button. Now the company is showing off two new concept laptops with rollable displays. As expected, the ThinkPad Rollable XD Concept is basically what you get if you take last […]
The post Lenovo’s rollable laptop concepts demonstrate new ways to use expandable displays appeared first on Liliputing.
The GMK EVO-T2 is a mini PC that looks a lot like last year’s EVO-T1. But by moving from an Intel Core Ultra 9 285H Arrow Lake processor to a new Core Ultra X9 388H Panther Lake chip, GMK says the new model brings up to a 50% boost in graphics performance, a more modest 10% improvement […]
The post GMK EVO-T2 is a mini PC with Intel Core Ultra X9 388H and up to 128GB LPDDR5x memory appeared first on Liliputing.
The Lenovo Legion Go 2 is a high-performance, versatile handheld gaming PC with an 8.8 inch FHD+ display, detachable controllers, and an AMD Ryzen Z2 Extreme processor with 16-core RDNA 3.5 graphics. When Lenovo first introduced the handheld in September, it was only available with Windows 11. But this summer the company plans to add a SteamOS […]
The post Lenovo Legion Go 2 with SteamOS and Ryzen Z2 Extreme coming this summer for $1200 appeared first on Liliputing.
The Motorola Razr Fold is a smartphone with a tablet-sized 8.1 inch OLED display that can fold in half like a book. Thanks to a 6.6 inch cover screen you can use it like a normal smartphone when folded or like a tablet when unfolded. In other words, it looks a lot like other foldable phones. […]
The post Motorola Razr Fold is coming later this year with 8.1 inch primary display and 6.6 inch cover screen appeared first on Liliputing.
CES 2026 appears to be the coming out party for WiFi 8 hardware… which is kind of weird because the WiFi 8 specification isn’t expected to be finalized for a few more years. But it’s not unusual to see companies start to release hardware for a work-in-progress version of the WiFi standard. What is a little […]
The post Lilbits: WiFi 8 hardware is coming this year, LEGO introduces SMART Brick system, and Intel could make handheld-specific Panther Lake chips appeared first on Liliputing.
The MINISFORUM AI X1 Pro is a versatile mini PC with support for up to 96GB of DDR5 memory (if you can afford it), up to three PCIe 4.0 NVMe SSDs and up to four displays. It has an OCuLink port for connecting an external graphics dock. And it features an AMD Ryzen AI 9 HX series […]
The post MINISFORUM AI X1 Pro mini PC get a Ryzen AI 9 HX 470 update appeared first on Liliputing.
The ONEXPLAYER Super V is an upcoming tablet with a 14 inch, 2880 x 1800 pixel OLED display, a detachable keyboard, and a powerful processor with high-performance graphics. If that all sounds familiar, that’s because it’s also true of the ONEXPLAYER Super X that hit Kickstarter in December. But while that model comes with up to AMD Ryzen […]
The post ONEXPLAYER Super V is a Panther Lake tablet with Intel Arc B390 graphics appeared first on Liliputing.
Intel may not be making NUC-branded mini PCs to showcase its latest mobile chips anymore, but Asus continues to carry the torch. A day after Intel officially launched its new Core Ultra Series 3 “Panther Lake” processor family, Asus has introduced a Panther Lake mini PC called the NUC 16 Pro. With support for up […]
The post Asus NUC 16 Pro mini PC supports up to Intel Core Ultra X9 388H with Arc B390 graphics appeared first on Liliputing.
Chip maker AMD is getting into the mini PC business… sort of. The new AMD Ryzen Halo is a compact desktop computer with AMD branding. But it’s not really designed for the general public. Instead, it’s positioned as an AI development platform. In other words, it’s the company’s answer to the NVIDIA DGX Spark, a mini […]
The post AMD Ryzen Halo is mini PC with Ryzen AI Max+ 395 and up to 128GB RAM (made for AI development) appeared first on Liliputing.
The Asus Zenbook Duo is a dual-screen laptop that features two full-sized displays, a wireless keyboard that can either be placed atop the bottom screen or positioned in front of the laptop giving you a full dual-screen workspace on the go. Asus introduced the first model with this basic design in 2024 and then gave […]
The post Asus Zenbook Duo (2026) minimizes the gap between screens appeared first on Liliputing.
The Asus TUF Gaming A14 (FA401EA) is a compact gaming laptop that weighs 3.26 pounds and measures 0.67 inches thick. Since it’s being marketed under the Asus TUF Gaming brand rather than the company’s ROG line of premium gaming PCs, it’d be easy to think of this as a cheaper alternative to the latest Asus ROG […]
The post Asus TUF Gaming A14 packs AMD Strix Halo into an 14 inch gaming laptop appeared first on Liliputing.
The Asus ExpertBook Ultra is a thin and light laptop with a 14 inch, 2880 x 1800 pixel, 120 Hz OLED touchscreen display, support for up to an Intel Core Ultra X9 388H Panther Lake processor and up to 64GB of LPDDR5X memory. While most laptops solder under the ExpertBook brand are made for business users, […]
The post Asus ExpertBook Ultra is an ultralight business laptop with a 2.8K OLED display and Intel Panther Lake appeared first on Liliputing.
The Asus Zenbook A16 is a laptop with a 16 inch, 2880 x 1800 pixel, 120 Hz OLED display. But weighing of just 1.3 kg (about 2.9 pounds), it’s lighter than most laptops with 13 or 14 inch displays, despite having a reasonably large 70 Wh battery inside the case. But that’s only one of the […]
The post Asus Zenbook A16 is a lightweight laptop with a 16 inch screen and Snapdragon X2 Elite Extreme appeared first on Liliputing.
The Khadas Mind line of mini PCs are compact desktop computer that could be used as standalone desktop computers that just happen to be small enough to fit in your pocket. But what makes them different from other mini PCs is a modular ecosystem that uses a PCIe 5.0 x8 expansion interface to let you connect […]
The post Khadas updates it modular mini PC ecosystem with new CPU, graphics dock, and display (plus a tablet… maybe) appeared first on Liliputing.
The Acer Swift Go 16 AI is a thin and light laptop with a big screen… and a choice of Intel or AMD processors. While the Intel model is a 3 pound notebook with support for up to an Intel Core Ultra X9 388H Panther Lake processor, the AMD version is a slightly heavier laptop […]
The post Acer’s new AMD-powered Swift Go 16 AI is a 3.5 pound OLED laptop with up to Ryzen AI 9 465 appeared first on Liliputing.
Backups on FreeBSD and Linux with restic: repositories and snapshots, backup and restore workflows, and using restic with AWS S3 and Google Drive via rclone.
Now you can signing up via the bot over ssh on Gothub.
My favorite albums from last year. Balkan brass, an acoustic favorite of 80s returns, Ethio-jazz, Guatemalan singer-guitarist, jazz-rock/Indian classical fusion, and a unique male vocalist.
Anthropic report on how their AI is changing their own software development practice.
❄ ❄ ❄ ❄ ❄
Much of the discussion about using LLMs for software development lacks details on workflow. Rather than just hear people gush about how wonderful it is, I want to understand the gritty details. What kinds of interactions occur with the LLM? What decisions do the humans make? When reviewing LLM outputs, what kinds of things are the humans looking for, what corrections do they make?
Obie Fernandez has written a post that goes into these kinds of details. Over the Christmas / New Year period he used Claude to build a knowledge distillation application, that takes transcripts from Claude Code sessions, slack discussion, github PR threads etc, turns them into an RDF graph database, and provides a web app with natural language ways to query them.
Not a proof of concept. Not a demo. The first cut of Nexus, a production-ready system with authentication, semantic search, an MCP server for agent access, webhook integrations for our primary SaaS platforms, comprehensive test coverage, deployed, integrated and ready for full-scale adoption at my company this coming Monday. Nearly 13,000 lines of code.
The article is long, but worth the time to read it.
An important feature of his workflow is relying on Test-Driven Development
Here’s what made this sustainable rather than chaotic: TDD. Test-driven development. For most of the features, I insisted that Claude Code follow the red-green-refactor cycle with me. Write a failing test first. Make it pass with the simplest implementation. Then refactor while keeping tests green.
This wasn’t just methodology purism. TDD served a critical function in AI-assisted development: it kept me in the loop. When you’re directing thousands of lines of code generation, you need a forcing function that makes you actually understand what’s being built. Tests are that forcing function. You can’t write a meaningful test for something you don’t understand. And you can’t verify that a test correctly captures intent without understanding the intent yourself.
The account includes a major refactoring, and much evolution of the initial version of the tool. It’s also an interesting glimpse of how AI tooling may finally make RDF useful.
❄ ❄ ❄ ❄ ❄
When thinking about requirements for software, most discussions focus on prioritization. Some folks talk about buckets such as the MoSCoW set: Must, Should, Could, and Want. (The old joke being that, in MoSCoW, the cow is silent, because hardly any requirements end up in those buckets.) Jason Fried has a different set of buckets for interface design: Obvious, Easy, and Possible. This immediately resonates with me: a good way of think about how to allocate the cognitive costs for those who use a tool.
❄ ❄ ❄ ❄ ❄
Casey Newton explains how he followed up on an interesting story of dark patterns in food delivery, and found it to be a fake story, buttressed by AI image and document creation. On one hand, it clarifies the important role reporters play in exposing lies that get traction on the internet. But time taken to do this is time not spent on investigating real stories
For most of my career up until this point, the document shared with me by the whistleblower would have seemed highly credible in large part because it would have taken so long to put together. Who would take the time to put together a detailed, 18-page technical document about market dynamics just to troll a reporter? Who would go to the trouble of creating a fake badge?
Today, though, the report can be generated within minutes, and the badge within seconds. And while no good reporter would ever have published a story based on a single document and an unknown source, plenty would take the time to investigate the document’s contents and see whether human sources would back it up.
The internet has always been full of slop, and we have always needed to be wary of what we read there. AI now makes it easy to manufacture convincing looking evidence, and this is never more dangerous than when it confirms strongly held beliefs and fears.
❄ ❄ ❄ ❄ ❄
The descriptions of Spec-Driven development that I have seen emphasize writing the whole specification before implementation. This encodes the (to me bizarre) assumption that you aren’t going to learn anything during implementation that would change the specification. I’ve heard this story so many times told so many ways by well-meaning folks–if only we could get the specification “right”, the rest of this would be easy.
Like him, that story has been the constant background siren to my career in tech. But the learning loop of experimentation is essential to the model building that’s at the heart of any kind of worthwhile specification. As Unmesh puts it:
Large Language Models give us great leverage—but they only work if we focus on learning and understanding. They make it easier to explore ideas, to set things up, to translate intent into code across many specialized languages. But the real capability—our ability to respond to change—comes not from how fast we can produce code, but from how deeply we understand the system we are shaping.
When Kent defined Extreme Programming, he made feedback one of its four core values. It strikes me that the key to making the full use of AI in software development is how to use it to accelerate the feedback loops.
❄ ❄ ❄ ❄ ❄
As I listen to people who are serious with AI-assisted programming, the crucial thing I hear is managing context. Programming-oriented tools are geting more sophisticated for that, but there’s also efforts at providing simpler tools, that allow customization. Carlos Villela recently recommended Pi, and its developer, Mario Zechner, has an interesting blog on its development.
So what’s an old guy yelling at Claudes going to do? He’s going to write his own coding agent harness and give it a name that’s entirely un-Google-able, so there will never be any users. Which means there will also never be any issues on the GitHub issue tracker. How hard can it be?
If I ever get the time to sit and really play with these tools, then something like Pi would be something I’d like to try out. Although as an addict to The One True Editor, I’m interested in some of libraries that work with that, such as gptel. That would enable me to use Emacs’s inherent programability to create my own command set to drive the interaction with LLMs.
❄ ❄ ❄ ❄ ❄
Outside of my professional work, I’ve posting regularly about my boardgaming on the specialist site BoardGameGeek. However its blogging environment doesn’t do a good job of providing an index to my posts, so I’ve created a list of my BGG posts on my own site. If you’re interested in my regular posts on boardgaming, and you’re on BGG you can subscribe to me there. If you’re not on BGG you can subscribe to the blog’s RSS feed.
I’ve also created a list of my favorite board games.
Gitanjali Venkatraman does wonderful illustrations of complex subjects (which is why I was so happy to work with her on our Expert Generalists article). She has now published the latest in her series of illustrated guides: tackling the complex topic of Mainframe Modernization
In it she illustrates the history and value of mainframes, why modernization is so tricky, and how to tackle the problem by breaking it down into tractable pieces. I love the clarity of her explanations, and smile frequently at her way of enhancing her words with her quirky pictures.
❄ ❄ ❄ ❄ ❄
Gergely Orosz on social media
Unpopular opinion:
Current code review tools just don’t make much sense for AI-generated code
When reviewing code I really want to know:
- The prompt made by the dev
- What corrections the other dev made to the code
- Clear marking of code AI-generated not changed by a human
Some people pushed back saying they don’t (and shouldn’t care) whether it was written by a human, generated by an LLM, or copy-pasted from Stack Overflow.
In my view it matters a lot - because of the second vital purpose of code review.
When asked why do code reviews, most people will answer the first vital purpose - quality control. We want to ensure bad code gets blocked before it hits mainline. We do this to avoid bugs and to avoid other quality issues, in particular comprehensibility and ease of change.
But I hear the second vital purpose less often: code review is a mechanism to communicate and educate. If I’m submitting some sub-standard code, and it gets rejected, I want to know why so that I can improve my programming. Maybe I’m unaware of some library features, or maybe there’s some project-specific standards I haven’t run into yet, or maybe my naming isn’t as clear as I thought it was. Whatever the reasons, I need to know in order to learn. And my employer needs me to learn, so I can be more effective.
We need to know the writer of the code we review both so we can communicate our better practice to them, but also to know how to improve things. With a human, its a conversation, and perhaps some documentation if we realize we’ve needed to explain things repeatedly. But with an LLM it’s about how to modify its context, as well as humans learning how to better drive the LLM.
❄ ❄ ❄ ❄ ❄
Wondering why I’ve been making a lot of posts like this recently? I explain why I’ve been reviving the link blog.
❄ ❄ ❄ ❄ ❄
Simon Willison describes how he uses LLMs to build disposable but useful web apps
These are the characteristics I have found to be most productive in building tools of this nature:
- A single file: inline JavaScript and CSS in a single HTML file means the least hassle in hosting or distributing them, and crucially means you can copy and paste them out of an LLM response.
- Avoid React, or anything with a build step. The problem with React is that JSX requires a build step, which makes everything massively less convenient. I prompt “no react” and skip that whole rabbit hole entirely.
- Load dependencies from a CDN. The fewer dependencies the better, but if there’s a well known library that helps solve a problem I’m happy to load it from CDNjs or jsdelivr or similar.
- Keep them small. A few hundred lines means the maintainability of the code doesn’t matter too much: any good LLM can read them and understand what they’re doing, and rewriting them from scratch with help from an LLM takes just a few minutes.
His repository includes all these tools, together with transcripts of the chats that got the LLMs to build them.
❄ ❄ ❄ ❄ ❄
Obie Fernandez: while many engineers are underwhelmed by AI tools, some senior engineers are finding them really valuable. He feels that senior engineers have an oft-unspoken mindset, which in conjunction with an LLM, enables the LLM to be much more valuable.
Levels of abstraction and generalization problems get talked about a lot because they’re easy to name. But they’re far from the whole story.
Other tools show up just as often in real work:
- A sense for blast radius. Knowing which changes are safe to make loudly and which should be quiet and contained.
- A feel for sequencing. Knowing when a technically correct change is still wrong because the system or the team isn’t ready for it yet.
- An instinct for reversibility. Preferring moves that keep options open, even if they look less elegant in the moment.
- An awareness of social cost. Recognizing when a clever solution will confuse more people than it helps.
- An allergy to false confidence. Spotting places where tests are green but the model is wrong.
❄ ❄ ❄ ❄ ❄
Emil Stenström built an HTML5 parser in python using coding agents, using Github Copilot in Agent mode with Claude Sonnet 3.7. He automatically approved most commands. It took him “a couple of months on off-hours”, including at least one restart from scratch. The parser now passes all the tests in html5lib test suite.
After writing the parser, I still don’t know HTML5 properly. The agent wrote it for me. I guided it when it came to API design and corrected bad decisions at the high level, but it did ALL of the gruntwork and wrote all of the code.
I handled all git commits myself, reviewing code as it went in. I didn’t understand all the algorithmic choices, but I understood when it didn’t do the right thing.
Although he gives an overview of what happens, there’s not very much information on his workflow and how he interacted with the LLM. There’s certainly not enough detail here to try to replicate his approach. This is contrast to Simon Willison (above) who has detailed links to his chat transcripts - although they are much smaller tools and I haven’t looked at them properly to see how useful they are.
One thing that is clear, however, is the vital need for a comprehensive test suite. Much of his work is driven by having that suite as a clear guide for him and the LLM agents.
JustHTML is about 3,000 lines of Python with 8,500+ tests passing. I couldn’t have written it this quickly without the agent.
But “quickly” doesn’t mean “without thinking.” I spent a lot of time reviewing code, making design decisions, and steering the agent in the right direction. The agent did the typing; I did the thinking.
❄ ❄
Then Simon Willison ported the library to JavaScript:
Time elapsed from project idea to finished library: about 4 hours, during which I also bought and decorated a Christmas tree with family and watched the latest Knives Out movie.
One of his lessons:
If you can reduce a problem to a robust test suite you can set a coding agent loop loose on it with a high degree of confidence that it will eventually succeed. I called this designing the agentic loop a few months ago. I think it’s the key skill to unlocking the potential of LLMs for complex tasks.
Our experience at Thoughtworks backs this up. We’ve been doing a fair bit of work recently in legacy modernization (mainframe and otherwise) using AI to migrate substantial software systems. Having a robust test suite is necessary (but not sufficient) to making this work. I hope to share my colleagues’ experiences on this in the coming months.
But before I leave Willison’s post, I should highlight his final open questions on the legalities, ethics, and effectiveness of all this - they are well-worth contemplating.
If you’re a regular reader of my site, you’ll have noticed that in the last few months I’ve been making a number of “fragments” posts. Such a post is a short post with a bunch of little, unconnected segments. These are usually a reference to something I’ve found on the web, sometimes a small thought of my own.
A few years ago, I wouldn’t have covered these topics with posts on my own site. Instead I would use Twitter, either retweeting someone else’s point, or just highlighting something I’d found. But since the Muskover, Twitter has effectively died. I’m not saying that due to any technical issues with the site, which has mostly just been fine, nor directly due to any of the policy changes there. The point is that lots of people have left, so that the audience I would have reached with Twitter is now fragmented. Some remain on X, but I see more activity on LinkedIn. There’s also Fediverse/Mastodon and Bluesky.
What this means for short posts is that I can no longer just post in one place. When I announce new articles on martinfowler.com, I announce now on four social media sites (X, LinkedIn, Fediverse, and Bluesky). It makes sense to do this, but I don’t want to go through all this hassle for the kind of micro-post that Twitter served so well.
So I’ve started to batch them up. When I see something interesting, I make a note. When I have enough notes, I post a fragments post. Initially I did this in a rather ad-hoc way, just using the same mechanisms I use for most articles, but last week I started to put in some more deliberate mechanisms into the site. (If you’re observant, you’ll spot that in the URLs.)
One benefit of all of this, at least in my book, is that it means my material is now fully visible in RSS. I’m probably showing my age, but I’m a big fan of RSS (or in my case, strictly Atom) feeds. I miss the feel of the heyday of the “blogosphere” before it got steamrolled by social media, and these fragment posts are, of course, just the same as the link blogs from that era. I still use my RSS reader every day to keep up with writers I like. (I’m pleased that Substack makes its content available via RSS.) It bothered me a bit that my micro-founts of Twitter knowledge weren’t visible on RSS, but was too lazy to do something about it. Now I don’t need to - the fragments are available in my RSS feed.
Why does AI write like… that (NYT, gift link). Sam Kriss delves into the quiet hum of AI writing. AI’s work is not compelling prose: it’s phantom text, ghostly scribblings, a spectre woven into our communal tapestry.
❄ ❄ ❄ ❄ ❄
Emily Bache has written a set of Test Desiderata, building on some earlier writing from Kent Beck. She lists the characteristics of good tests, and how they support her four “macro desiderata” - the properties of a sound test suite
- Predict success in production
- Fast to get feedback
- Support ongoing code design change
- Low total cost of ownership
She also has a great list of other writers’ lists of good test characteristics.
❄ ❄ ❄ ❄ ❄
Daphne Keller explains that the EUs fines on X aren’t about free speech.
There are three charges against X, which all stem from a multi-year investigation that was launched in 2023. One is about verification — X’s blue checkmarks on user accounts — and two are about transparency. These charges have nothing to do with what content is on X, or what user speech the platform should or should not allow.
❄ ❄ ❄ ❄ ❄
Cory Doctorow The Reverse-Centaur’s Guide to Criticizing AI
Start with what a reverse centaur is. In automation theory, a “centaur” is a person who is assisted by a machine. … And obviously, a reverse centaur is machine head on a human body, a person who is serving as a squishy meat appendage for an uncaring machine.
Like an Amazon delivery driver… the van can’t drive itself and can’t get a parcel from the curb to your porch. The driver is a peripheral for a van, and the van drives the driver, at superhuman speed, demanding superhuman endurance.
Rob Bowley summarizes a study from Carnegie Mellon looking on the impact of AI on a bunch of open-source software projects. Like any such study, we shouldn’t take its results as definitive, but there seems enough there to make it a handy data point. The key point is that the AI code probably reduced the quality of the code base - at least if static code analysis can be trusted to determine quality. And perhaps some worrying second-order effects
This study shows more than 800 popular GitHub projects with code quality degrading after adopting AI tools. It’s hard not to see a form of context collapse playing out in real time. If the public code that future models learn from is becoming more complex and less maintainable, there’s a real risk that newer models will reinforce and amplify those trends, producing even worse code over time.
❄ ❄ ❄ ❄ ❄
Rob’s post is typical of much of the thoughtful writing on AI. We can see its short-term benefits, but worry about its long-term impact. But on a much deeper note is this lovely story from Jim Highsmith. Jim has turned 0x50, and has spent the last decade fighting Parkinson’s disease. To help him battle it he has two AI assisted allies.
Between my neural implants and Byron’s digital guidance, I now collaborate with two adaptive systems: one for motion, one for thought. Neither replaces me. Both extend me.
If you read anything on AI this week, make it be this. It offers a positive harbinger for our future and opens my mind to a whole different perspective of the role of AI in it
❄ ❄ ❄ ❄ ❄
Anthropic recently announced that it disrupted a Chinese state-sponsored operation abusing Claude Code. Jim Gumbley looks at the core lesson to learn from this, that we have to understand the serious risk of AI Jailbreaking
New AI tools are able to analyze your attack surface at the next level of granularity. As a business leader, that means you now have two options: wait for someone else to run AI-assisted vulnerability detection against your attack surface, or run it yourself first.
❄ ❄ ❄ ❄ ❄
There’s plenty of claims that AI Vibe Coding can replace software developers, something that folks like me (perhaps with a bias) think unlikely. Gergely Orosz shared this tidbit
Talked with an exec at a tech company who is obsessed with AI and has been for 3 years. Not a developer but company makes software. Uses AI for everything, vibe codes ideas.
Here’s the kicker:
Has a team of several devs to implement his vibe coded prototypes to sg workable
I’d love to hear more about this (and similar stories)
❄ ❄ ❄ ❄ ❄
Nick Radcliffe writes about a month of using AI
I spent a solid month “pair programming” with Claude Code, trying to suspend disbelief and adopt a this-will-be-productive mindset. More specifically, I got Claude to write well over 99% of the code produced during the month. I found the experience infuriating, unpleasant, and stressful before even worrying about its energy impact. Ideally, I would prefer not to do it again for at least a year or two. The only problem with that is that it “worked”.
He stresses that his approach is the “polar opposite” of Vibe Coding. The post is long, and rambles a bit, but is worthwhile because he talks in detail about his workflow and how he uses the tool. Such posts are important so we can learn the nitty-gritty of how our programming habits are changing.
❄ ❄ ❄ ❄ ❄
Along similar lines is a post of Brian Chambers on his workflow, that he calls Issue-Driven Development (and yes, I’m also sick of the “something-driven” phraseology). As with much of the better stuff I’ve heard about AI assisted work, it’s all about carefully managing the context window, ensuring the AI is focused on the right things and not distracted by textual squirrels.
I’ve been on the road in Europe for the last couple of weeks, and while I was there Thoughtworks released volume 33 of our Technology Radar. Again it’s dominated by the AI wave, with lots of blips capturing our explorations of how to use LLMs and similar technology. “Agents” are the big thing these days but we’re also seeing growing movements in infrastructure orchestration, coding workflows - and the inevitable antipatterns. Many thanks to my colleagues for putting this together again.
❄ ❄ ❄ ❄
My trip to Europe started in Amsterdam, for a Thoughtworks event for a few of our clients there. Since I was in that lovely city, I got in touch with Gergely Orosz, host of The Pragmatic Engineer, and he arranged to record a podcast with me. No surprise that AI was front-and-center of the conversation, as I said it was the biggest shift I’d seen in programming during my career, comparable only to the shift to high-level languages, which even I am not old enough to have experienced. It was a fun chat and I really enjoyed myself. Gergely later joined myself James Lewis and Giles Edwards-Alexander at the Thoughtworks event the next day.
❄ ❄ ❄ ❄
My travels also took me to Nüremberg, where I attended an internal conference for Siemens on the future of software architecture. When we think of technology, it’s easy to focus on the Faangs of Silicon Valley, but Siemens have a huge workforce of software developers working on heavy engineering systems like trains and factory automation. It was good to hear them talk about federated architectures, data mesh, and their use of AI.
❄ ❄ ❄ ❄
I’ve often used pseudo-graphs to help explain why high quality software is cheaper. This time, Kent Beck creates a unique perspective to this chart, dispensing with the temporal axis to help think in terms of optionality.
❄ ❄ ❄ ❄
And in another life, Edward has finally finished the great migration of the Heavy Cardboard studio and returns to the tubes with our first game in the new digs. (No surprise that it’s Age of Steam.)
I find most writing on software productivity to be twaddle, but Nicole Forsgren and Abi Noda are notable exceptions. I had a chance to take a look at their new book, published today, and liked it so much I wrote a foreword.
Unmesh Joshi finds LLMs to be a useful tool, but explains why their help becomes illusory if we use them to shortcut the learning loop that's an essential part of our professional practice.
I’m very concerned about the security dangers of LLM-enabled browsers, as it’s just too easy for them to contain the Lethal Trifecta. For up-to-date eyes on these issues, I follow the writings of coiner of that phrase: Simon Willison. Here he examines a post on how OpenAI is thinking about these issues.
My takeaways from all of this? It’s not done much to influence my overall skepticism of the entire category of browser agents, but it does at least demonstrate that OpenAI are keenly aware of the problems and are investing serious effort in finding the right mix of protections.
❄ ❄ ❄ ❄
Unsurprisingly, there are a lot of strong opinions on AI assisted coding. Some engineers swear by it. Others say it’s dangerous. And of course, as is the way with the internet, nuanced positions get flattened into simplistic camps where everyone’s either on one side or the other.
A lot of the problem is that people aren’t arguing about the same thing. They’re reporting different experiences from different vantage points.
His view is that beginners are very keen on AI-coding but they don’t see the problems they are creating. Experienced folks do see this, but it takes a further level of experience to realize that when used well these tools are still valuable.
Interestingly, I’ve regularly seen sceptical experienced engineers change their view once they’ve been shown how you can blend modern/XP practices with AI assisted coding.
The upshot is this, is that you have be aware of the experience level of whoever is writing about this stuff - and that experience is not just in software development generally, but also in how to make use of LLMs. One thing that rings clearly from reading Simon Willison and Birgitta Böckeler is that effective use of LLMs is a skill that takes a while to develop.
❄ ❄ ❄ ❄
Charlie Brown and Garfield, like most comic strip characters, never changed over the decades. But Doonesbury’s cast aged, had children, and some have died (I miss Lacey). Gary Trudeau retired from writing daily strips a few years ago, but his reruns of older strips is one of the best things in the shabby remains of Twitter.
A couple of weeks ago, he reran one of the most memorable strips in its whole run. The very first frame of Doonesbury introduced the character “B.D.”, a football jock never seen without his football helmet, or when on duty, his military helmet. This panel was the first time in over thirty years that B.D. was shown without a helmet, readers were so startled that they didn’t immediately notice that the earlier explosion had removed his leg. This set off a remarkable story arc about the travails of a wounded veteran. It’s my view that future generations will find Doonesbury to be a first-class work of literature, and a thoughtful perspective on contemporary America.
Agentic AI systems are amazing, but introduce equally amazing security risks. Korny Sietsma explains that their core architecture opens up security issues through what Simon Willison named the “Lethal Trifecta”. Korny goes on to talk about how to mitigate this through removing legs of the trifecta and splitting complex tasks.
Mathias Verraes writes about the relationship between Domains and Bounded Contexts in Domain-Driven Design. It’s a common myth that there should always be a 1:1 relationship between them, but although it’s sometimes the case, deeper modeling often exposes a more interesting structure.
Gary Marcus: (NYT Gift Link)
If the strengths of A.I. are truly to be harnessed, the tech industry should stop focusing so heavily on these one-size-fits-all tools and instead concentrate on narrow, specialized A.I. tools engineered for particular problems. Because, frankly, they’re often more effective.
One of the truly annoying things about the US tax system is that we can’t easily file our tax returns electronically. In recent years an initiative called “Direct File” sought to fix that. Matt Bracken tells the story of how they developed a highly-regarded system in 25 states, but was canned by the Trump administration. He also explains how the creators of Direct File are working to prepare the ground for it to reappear.
Security issues are only getting worse, but the US government agency for cybersecurity is having its staff reassigned to other duties. Detailed story in Bloomberg (paywalled) and an open (but more polemic) summary on Techdirt.
Changes have hit particularly hard in CISA’s Capacity Building team, which writes emergency directives and oversees cybersecurity for the government’s highest value assets, the employees said.
Defense and law enforcement are valuable things for a government to do, but here they seem to be walking away from a growing crisis.
Birgitta Böckeler has been trying to understand one of the latest AI coding buzzword: Spec-driven development (SDD). She looked at three of the tools that label themselves as SDD tools and tried to untangle what it means, as of now.
Service templates are a typical building block in the “golden paths” organisations build for their engineering teams, to make it easy to do the right thing. The templates are supposed to be the role models for all the services in the organisation, always representing the most up to date coding patterns and standards. One of the challenges with service templates though is that once a team instantiated a service with one, it’s tedious to feed template updates back to those services. Birgitta Böckeler considers whether GenAI can help with that.
Birgitta Böckeler examines the risk assessment around when to use vibe coding, using three dimensions of risk: Probability, Impact, and Detectability
I’m about to head away from looking after this site for a few weeks (part vacation, part work stuff). As I contemplate some weeks away from the daily routine, I feel an urge to share some scattered thoughts about the state of LLMs and AI.
❄ ❄ ❄ ❄
I’ve seen a few early surveys on the effect AI is having on software development, is it really speeding folks up, does it improve or wreck code quality? One of the big problems with these surveys is that they aren’t taking into account how people are using the LLMs. From what I can tell the vast majority of LLM usage is fancy auto-complete, often using co-pilot. But those I know who get the most value from LLMs reckon that auto-complete isn’t very useful, preferring approaches that allow the LLM to directly read and edit source code files to carry out tasks. My concern is that surveys that ignore the different work-flows of using LLMs will produce data that’s going to send people down the wrong paths.
(Another complication is the varying capabilities of different models.)
❄ ❄ ❄ ❄
I’m often asked, “what is the future of programming?” Should people consider entering software development now? Will LLMs eliminate the need for junior engineers? Should senior engineers get out of the profession before it’s too late? My answer to all these questions is “I haven’t the foggiest”. Furthermore I think anyone who says they know what this future will be is talking from an inappropriate orifice. We are still figuring out how to use LLMs, and it will be some time before we have a decent idea of how to use them well, especially if they gain significant improvements.
What I suggest, is that people experiment with them. At the least, read about what others are doing, but pay attention to the details of their workflows. Preferably experiment yourself, and do share your experiences.
❄ ❄ ❇ ❄
I’m also asked: “is AI a bubble”? To which my answer is “OF COURSE IT’S A BUBBLE”. All major technological advances have come with economic bubbles, from canals and railroads to the internet. We know with near 100% certainty that this bubble will pop, causing lots of investments to fizzle to nothing. However what we don’t know is when it will pop, and thus how big the bubble will have grown, generating some real value in the process, before that happens. It could pop next month, or not for a couple of years.
We also know that when the bubble pops, many firms will go bust, but not all. When the dot-com bubble burst, it killed pets.com, it killed Webvan… but it did not kill Amazon.
❄ ❄ ❄ ❄
I retired from public speaking a couple of years ago. But while I don’t miss the stress of giving talks, I do miss hanging out with my friends in the industry. So I’m looking forward to catching up with many of them at GOTO Copenhagen. I’ve been involved with the GOTO conference series since the 1990s (when it was called JAOO), and continue to be impressed with how they put together a fascinating program.
✢ ❄ ❄ ❄
My former colleague Rebecca Parsons, has been saying for a long time that hallucinations aren’t a bug of LLMs, they are a feature. Indeed they are the feature. All an LLM does is produce hallucinations, it’s just that we find some of them useful.
One of the consequences of this is that we should always consider asking the LLM the same question more than once, perhaps with some variation in the wording. Then we can compare answers, indeed perhaps ask the LLM to compare answers for us. The difference in the answers can be as useful as the answers themselves.
Certainly if we ever ask a hallucination engine for a numeric answer, we should ask it at least three times, so we get some sense of the variation. Furthermore we shouldn’t ask an LLM to calculate an answer than we can calculate deterministically (yes, I’ve seen this). It is OK to ask an LLM to generate code to calculate an answer (but still do it more than once).
❄ ❄ ❄ ❄
Other forms of engineering have to take into account the variability of the world. A structural engineer builds in tolerance for all the factors she can’t measure. (I remember being told early in my career that the unique characteristic of digital electronics was that there was no concept of tolerances.) Process engineers consider that humans are executing tasks, and will sometimes be forgetful or careless. Software Engineering is unusual in that it works with deterministic machines. Maybe LLMs mark the point where we join our engineering peers in a world on non-determinism.
❄ ❄ ❄ ❄
I’ve often heard, with decent reason, an LLM compared to a junior colleague. But I find LLMs are quite happy to say “all tests green”, yet when I run them, there are failures. If that was a junior engineer’s behavior, how long would it be before H.R. was involved?
❄ ❄ ❄ ❄
LLMs create a huge increase in the attack surface of software systems. Simon Willison described the The Lethal Trifecta for AI agents: an agent that combines access to your private data, exposure to untrusted content, and a way to externally communicate (“exfiltration”). That “untrusted content” can come in all sorts of ways, ask it to read a web page, and an attacker can easily put instructions on the website in 1pt white-on-white font to trick the gullible LLM to obtain that private data.
This is particularly serious when it comes to agents acting in a browser. Read an attacker’s web page, and it could trick the agent to go to your bank account in another tab and “buy you a present” by transferring your balance to the kind attacker. Willison’s view is that “the entire concept of an agentic browser extension is fatally flawed and cannot be built safely”.
A common enterprise problem: crucial legacy systems become “black boxes”—key to operations but opaque and risky to touch. Thiyagu Palanisamy and Chandirasekar Thiagarajan worked with a client to use AI-assisted reverse engineering to reconstruct functional specifications from UI elements, binaries, and data lineage to overcome analysis paralysis. They developed a methodical “multi-lens” approach—starting from visible artifacts, enriching incrementally, triangulating logic, and always preserving lineage. Human validation remains central to ensure accuracy and confidence in extracted functionality. This engagement revealed that turning a system from black box to blueprint empowers modernization decisions and accelerates migration efforts.
The Bahmni open-source hospital management system was began over nine years ago with a front end using AngularJS and an OpenMRS REST API. Rahul Ramesh wished to convert this to use a React + TypeScript front end with an HL7 FHIR API. In exploring how to do this modernization he used a structured prompting workflow of Research, Review, and Rebuild - together with Cline, Claude 3.5 Sonnet, Atlassian MCP server, and a filesystem MCP server. Changing a single control would normally take 3–6 days of manual effort, but with these tools was completed in under an hour at a cost of under $2.
CLI coding agents are a fundamentally different tool to chatbots or autocomplete tools - they're agents that can read code, run tests, and update a codebase. Ben O'Mahony explains that while commercial tools are impressive, they don't understand the particular context of our environment and the eccentricities of our specific project. Instead we can build our own coding agent by assembling open source tools, using our specific development standards for: testing, documentation production, code reasoning, and file system operations.
A few weeks ago, Unmesh Joshi and I started having a conversation about how he likes to grow a language of abstractions when working with an LLM. We thought this was a conversation that others might find interesting so we turned it into an article. We talk about how programming is about both building and applying abstractions and how the LLM helps us in different ways with each activity.
Back in the days when I did live talks, one of my abilities was to finish on time, even if my talk time was cut at the last moment (perhaps due to the prior speaker running over). The key to my ability to do this was to use Expansion Joints - parts of the talk that I'd pre-planned so I could cover them quickly or slowly depending on how much time I had.
The way I'd do this would be to plan for some topics to be optional. The talk would work if I skipped over them, but I could also witter on about them for five (or ten) minutes. Ideally, each of these topics would get one slide, usually with a bunch of key phrases on it - the headings of what I'd talk about should I be talking about it. When I got to the slide, I'd look at how time was going with the talk. If (as was usually the case) I was running short of time, I could cover the slide in about thirty seconds, saying something like: “in doing this, there's a bunch of things you need to consider, but they are out of scope for today's talk”.
If, however, I did have time, I could then spend some time talking about them. The slide would be simple, and not provide much of a Visual Channel, but that wasn't so important, after all this material was optional in the first place.
The single flex-slide was my favorite Expansion Joint, as it was easy to use. Sometimes however my optional topic required a proper visual channel, necessitating dedicated slides. My solution here was good control over slide handling. Presentation tools include the ability to skip over slides while I'm talking, and I made sure I practiced how to use them so I could skip a bunch of slides without the audience knowing. It's crucial here that it's invisible to the audience, I find it looks sloppy if anyone says “in the interests of time I'll skip over these slides”. To do this, however, I do need access to my laptop while presenting, venues that only provide a clicker while loading the slides on some other machine lack that control. That started to happen in my last couple of years, much to my annoyance.
When creating talks, I was always worried that I would run out of things to say, even though experience told me I reliably crammed more stuff in than I could possibly cover. Expansion Joints helped with this, I could aggressively trim the core talk to less than I needed, and rely on the Expansion Joints to fill the gap. In practice I usually didn't need the Expansion Joints anyway, but their presence helped my confidence.
Using Expansion Joints was particularly important for me as I never rehearsed my talks. I was always someone whose ability to present was driven by adrenaline. Talking to a rubber duck just didn't work, the duck was clearly every bit as bored as I was. Consequently the first time I gave a talk, I was hazy as to how long it would take. Yet with Expansion Joints in place, I was able to finish a talk right on time.
Expansion Joints enabled me to give the same talk to different time slots. Sometimes I'd have thirty minutes, sometimes forty-five. With Expansion Joints, I didn't need to change my slides, particularly handy if a time cut (or more rarely a time increase) appeared at the last moment. (Although in my later years, I handled this by doing a Suite Of Talks.)
Talks that encourage audience interaction need these because we can never predict how much time the interaction will use up. Sometimes we get a steady stream of questions, other times (particularly in Scandinavia, or upper-Midwest America) a lack of questions had me blasting through the agenda. Any such talk needed a double-dose of this temporal ballast.
Expansion Joints are at their most useful in later parts of the talk, as it's then that I have the most information on how much time I have. Earlier ones can still be handy, particularly if they come after an interactive section when I'd like to rebase my timing.
The name was coined by Neal Ford, Matthew McCullough, and Nathaniel Schutta in their excellent book Presentation Patterns.
OKRs have become a popular way to connect strategy with execution in large organizations. But when they are set in a top‑down cascade, they often lose their meaning. Teams receive objectives they didn’t help create, and the result is weak commitment and little real change. Paulo Caroli describes how high‑performing teams can work in another way. They define their own objectives in an organization that uses a collaborative process to align the team’s OKRs with the broader strategy. With these Team OKRs in place, they create a shared purpose and become the base for a regular cycle of planning, check‑ins, and retrospectives.
Sriram Narayan concludes his article in impact intelligence by addressing five common objections to this activity, including slowing down, lack of agility and collaboration, and the unpredictability of innovation.
Abi Noda observes
Just met with a 2000+ eng company. Their developers are saving 2+ hours per week thanks to Copilot.
But they’re also losing:
- 3 hrs per week due to slow builds
- 4 hrs per week on dev environment toil
- 2 hrs per week waiting for code reviews
AI is not a silver bullet.
Nik Malykhin found it useful to get an AI assistant to write its own coding rules by analyzing his code, and then asking it to refine them as worked with it.
the central paradox of using AI assistants effectively: to offload cognitive work to an AI, you must first do the meta-cognitive work of codifying your own development philosophy and collaboration style.
I agree with Charity Majors that there is a valuable distinction between disposable versus durable code, and that makes a difference in how we use AI with it.
The difference between disposable code and durable code is not about whether the code was generated by AI or written by a human, or even how difficult it was to write. The cost is defined by the standards you are building to, and the rest of the software development lifecycle: how well you expect to maintain it, extend it, migrate it, understand its behavior, or fix it when it breaks. This is the expensive part of software development, the type that requires deep expertise and familiarity with your language and environment. Disposable code is cheap because you don’t even try to maintain it.
Jim Highsmith thinks that we should think of AI as Alternative Intelligence
It’s not fake intelligence, or artificial empathy, or HAL 9000 with manners. It’s something else. Something that thinks differently, not defectively.
Rod Johnson asserts that we know that memory is important to AI systems, but we forget that Domain Models are an important form of memory
Event Sourcing provides perfect episodic memory by storing the complete history of domain changes as immutable events. Every decision, every state transition, every business event is preserved with full context.
Repository patterns offer domain-focused memory interfaces that understand business concepts. A CustomerRepository knows how to retrieve customer information in ways that preserve business meaning, not just raw data.
Bounded contexts from Domain-Driven Design partition memory into semantic boundaries, preventing the concept pollution that plagues pure vector-based approaches.
Aggregates function as cohesive memory clusters with consistency boundaries—exactly what we need for reliable agent behavior.
Sriram Narayan continues his article on impact intelligence by outlining five actions that can be done to improve impact intelligence: introduce robust demand management, pay down measurement debt introduce impact validation, offer your CFO/COO an alternative to ROI, equip your teams.
The productivity of knowledge workers is hard to quantify and often decoupled from direct business outcomes. The lack of understanding leads to many initiatives, bloated tech spend, and ill-chosen efforts to improve this productivity. Sriram Narayan begins an article that looks at how to avoid this by developing an intelligence of the business impact of their work across a network connecting output to proximate and downstream impact.
Birgitta Böckeler reports on a series of experiments we did to explore how far Generative AI can currently be pushed toward autonomously developing high-quality, up-to-date software without human intervention. As a test case, we created an agentic workflow to build a simple Spring Boot application end to end. We found that the workflow could ultimately generate these simple applications, but still observed significant issues in the results—especially as we increased the complexity. The model would generate features we hadn't asked for, make shifting assumptions around gaps in the requirements, and declare success even when tests were failing. We concluded that while many of our strategies — such as reusable prompts or a reference application — are valuable for enhancing AI-assisted workflows, a human in the loop to supervise generation remains essential.
Matteo Vaccari shows why the common metric of AI code acceptance has big hole. An LLM can be helpful even if you throw away its code.
It's become a common habit for developers to give Large Language Models (LLMs) a persona when working with them. I describe four of them, a stubborn donkey, a genie, a slot machine, and Uriah Heep.
Korny Sietsma has a great example of how using an LLM for coding is very helpful but with limitations…
and a thoughtful general essay on why the hype and the immovable skeptics are both missing the train.
While here, a professor of poetry ponders (gift link) on the value and limits of AI with writing:
One of the real challenges here is the way that A.I. undermines the human value of attention, and the individuality that flows from that.
What we stand to lose is not just a skill but a mode of being: the pleasure of invention, the felt life of the mind at work.
The IEA's special report Energy and AI, out today, offers the most comprehensive, data-driven global analysis to date on the growing connections between energy and AI. The report draws on new datasets and extensive consultation with policy makers, the tech sector, the energy industry and international experts. It projects that electricity demand from data centres worldwide is set to more than double by 2030 to around 945 terawatt-hours (TWh), slightly more than the entire electricity consumption of Japan today. AI will be the most significant driver of this increase, with electricity demand from AI-optimised data centres projected to more than quadruple by 2030.
The Arduino GitHub has been busy. Over the past few weeks, we’ve published and updated dozens of repositories spanning libraries, cores, and development tools, and we want you to explore them. Whether you’re working with our newest hardware, building with the Arduino IDE, or diving into platform development, there’s something new for you to discover, […]
The post Explore the latest: Arduino’s growing open source ecosystem appeared first on Arduino Blog.
Designed by Cirket open-source hardware in China, the ESP32-DIV V1 is a handheld wireless pentesting and experimentation tool with Wi-Fi, BLE, generic 2.4 GHz (NRF24), and Sub-GHz RF (CC1101) communication. The device targets hardware hackers and cybersecurity researchers for learning and testing wireless vulnerabilities across multiple frequency bands. The device features a modular “sandwich” design consisting of a Main Board and a Shield Board connected via a 20-pin header. The Main Board integrates an ESP32 microcontroller, a 2.8-inch ILI9341 TFT display with XPT2046 touch controller, SD card slot, battery charging and power management, and navigation controls. The Shield Board hosts the RF hardware, including three NRF24L01 modules, a CC1101 Sub-GHz transceiver, and multiple SMA antenna connectors. ESP32-DIV (V1) specifications: Wireless Module – ESP32-WROOM-32U SoC – ESP32 dual-core wireless microcontroller CPU – Dual-core Xtensa 32-bit microprocessor @ 240MHz Memory – 520KB internal SRAM Wireless – Wi-Fi 802.11b/g/n, and Bluetooth (4.2 and [...]
The post ESP32-DIV V1 handheld pentesting tool supports Wi-Fi Attacks, BLE spoofing, 2.4GHz scanning, and Sub-GHz jamming appeared first on CNX Software - Embedded Systems News.
Bottle prepping machines make life just a little bit less hectic for overwhelmed parents, but they do still need to be started and that is easier said than done when you’ve got a wailing baby in your arms. That’s why Manivannan created this edge AI-powered system that automatically starts a bottle prepping machine when it […]
The post AI automatically preps a bottle when the baby cries appeared first on Arduino Blog.
Every millennial knows the exhilarating feeling of going to the computer lab, booting up a Macintosh, and creating beautiful art in MacPaint. The nostalgia meter has broken its dial and is now spinning wildly. If you want to capture that nostalgia in a form suitable for home décor, you can build Mark Wilson’s fantastic LackPaint […]
The post Display your photos like they’re in MacPaint appeared first on Arduino Blog.
Every year, some of YouTube’s most prolific makers get together to coordinate a Secret Santa exchange among themselves, then post videos of the results. The only rule is that they have to create the gifts. This year, Matty Benedetto (Unnecessary Inventions) got Austin Bradley in the exchange. For Bradley’s present, Benedetto constructed this set of […]
The post A super-sized set of digital calipers for giants appeared first on Arduino Blog.
M5Stack NanoH2 is an ultra-compact, standalone IoT development kit powered by an Espressif Systems’ ESP32-H2 wireless SoC with support for Zigbee, Thread, and Matter, and designed for smart home and low-power wireless applications. It also features a built-in LED, an RGB LED, an infrared transmitter, a Grove port for expansion, a user button, and a USB-C port for power and programming. On the surface, it looks very similar to the Unit Gateway H2 introduced last March, with an ESP32-H2 SoC and a Grove connector. But while the earlier model is designed as an add-on board to other M5Stack products, the NanoH2 is a standalone development kit that you can use directly without another board from M5Stack. M5Stack NanoH2 specifications: SoC – Espressif Systems ESP32-H2FH4S CPU – 32-bit RISC-V SoC clocked at up to 96 MHz Memory – 320 KB SRAM, 4 KB LP memory Storage – 128 KB ROM, 4MB [...]
The post M5Stack NanoH2 – An ultra-compact ESP32-H2 IoT development kit with Zigbee, Thread, and Matter connectivity appeared first on CNX Software - Embedded Systems News.
Brian got his hands on an old Lorenz 15 teletype machine, which was made in the 1950s. He also managed to snag an appropriate teletype modem. But the modem couldn’t seem to understand recorded teletype messages, so Brian used an Arduino Mega 2560 to “speak” teletype and get the Lorenz 15 printing. You can think […]
The post Arduino speaks teletype to an old Lorenz 15 appeared first on Arduino Blog.
For the last five years, once a month, we have hosted an online seminar sharing computing education research. Seminars are organised as usually year-long series with changing themes. In 2025, for example, our theme was ‘Teaching about AI and data science’. In 2024, it was ‘Teaching programming (with or without AI)’. It is not surprising…
The post How can we teach about AI in the arts, humanities and sciences? Research seminar series 2026 appeared first on Raspberry Pi Foundation.
To mark the start of 2026, we’re releasing a special episode of our Hello World podcast, which reflects on the key developments in computing education during 2025 and considers the trends likely to shape the year ahead. Hosted by James Robinson, the episode brings together a conversation between three Foundation team members — Rehana Al-Soltane,…
The post What shaped computing education in 2025 — and what comes next appeared first on Raspberry Pi Foundation.
Research indicates that teaching learners to use and create with data-driven technologies such as AI and machine learning (ML) requires an entirely different approach for solving problems compared to traditional programming activities. In this blog, we share the new data paradigms framework that we have developed through research and used to help improve our understanding…
The post A research-led framework for teaching about models in AI and data science appeared first on Raspberry Pi Foundation.
In the age of AI, we believe kids still need to learn to code. We’re also testing how AI tools can be used to help with teaching and learning programming. This blog dives into our new classroom unit ‘Programming with AI’, which we’ve developed for teachers to introduce students to learning to code using tools…
The post Introducing our new ‘Programming with AI’ unit appeared first on Raspberry Pi Foundation.
Today, we are marking the 10th anniversary of the European Astro Pi Challenge. At 11:03 on 15 December 2015, former ESA Astronaut Tim Peake launched from Baikonur cosmodrome in Kazakhstan on a Soyuz rocket as part of his UK Space agency ‘Principia’ mission to the International Space Station (ISS). Two Raspberry Pi-powered Astro Pi Mark…
The post Astro Pi is 10: A decade of your code in space appeared first on Raspberry Pi Foundation.
It’s been over a year since I last wrote an update on this blog about our research and as we’ve just published our 2025 Annual Report, this is an ideal opportunity to share what we’ve been working on at the Raspberry Pi Computing Education Research Centre. We are a research centre based in the Department…
The post 2025 highlights from the Raspberry Pi Computing Education Research Centre appeared first on Raspberry Pi Foundation.
Two remarkable teenagers from Pune in the state of Maharashtra, India — Jayantika (age 16) and Ruturaj (age 14) — have turned curiosity into community impact. With support from the Bidkar Foundation and the Raspberry Pi Foundation, they now help run five Code Clubs, reaching over 200 young people in rural areas of Pune. Along…
The post From learners to leaders: Jayantika and Ruturaj’s journey as Code Club youth mentors appeared first on Raspberry Pi Foundation.
At a time when many young people are using AI for personal and learning purposes, schools are trying to figure out what to teach about AI and how (find out more in this summer 2025 data about young people’s usage of AI in the UK). One aspect of this is how technical we should get…
The post Secondary school maths showing that AI systems don’t think appeared first on Raspberry Pi Foundation.
We love hearing from members of the community and sharing the stories of amazing young people, volunteers, and educators who are using their passion for technology to create positive change in the world around them. In our latest story, we’re meeting a group of inspiring young innovators from Belfast — Michelle, 15, Inilouwa, 18, Jedidiah,…
The post Celebrating the community: Irioluwa, Michelle, Jedidiah and Inioluwa appeared first on Raspberry Pi Foundation.
Listening to the voices of young people is crucial in creating genuinely effective learning resources. That’s why we recently ran a survey of students who use Ada Computer Science, our platform designed to help students learn and revise key computer science concepts. “The different topics are nicely categorised and it is easy to find the…
The post How Ada Computer Science empowers students: Survey findings appeared first on Raspberry Pi Foundation.
For a list of changes please read the CHANGES file; for release notes please see this issue
For a list of changes please read the CHANGES file; for release notes please see this issue
This is a bug fix release of 3.5
For a list of changes please read the CHANGES file; for release notes please see this issue
For a list of changes please read the CHANGES file; for release notes please see this issue
For a list of changes please read the CHANGES file; for release notes please see this issue
This is a bug fix release of 3.3
For a list of changes please read the CHANGES file; for release notes please see this issue
For a list of changes please read the CHANGES file; for release notes please see this issue
This is a bug fix release of 3.2
For a list of changes please read the CHANGES file; for release notes please see this issue
For a list of changes please read the CHANGES file; for release notes please see this issue
This is a bug fix release of 3.1
For a list of changes please read the CHANGES file
Mac Tahoe (in Beta as of the time of this writing) has this new feature called Edge Light that basically puts a bright picture of an Edge Light around your screen and basically uses the power of OLED to give you a virtual ring light. So I was like, why can't we also have nice things? I wrote (vibed, with GitHub Copilot and Claude Sonnet 4.5) a Windows Edge Light App (source code at https://github.com/shanselman/WindowsEdgeLight and you can get the latest release here https://github.com/shanselman/WindowsEdgeLight/releases or the app will check for new releases and autoupdate with Updatum).
However, as is with all suss loose executables on the internet, when you run random stuff you'll often get the Window Defender 'new phone, who dis' warning which is scary. After several downloads and no viruses or complaints, my executable will eventually gain reputation with the Windows Defender Smart Screen service, but having a Code Signing Certificate is said to help with that. However, code signing certs are expensive and a hassle to manage and renew.
Someone told me that Azure Trusted Signing was somewhat less of a hassle - it's less, but it's still non-trivial. I read this post from Rick (his blog is gold and has been for years) earlier in the year and some of it was super useful and other stuff has been made simpler over time.
I wrote 80% of this blog post, but since I just spent an hour getting code signing to work and GitHub Copilot was going through and logging everything I did, I did use Claude 4.5 to help organize some of this. I have reviewed it all and re-written parts I didn't like, so any mistakes are mine.
Azure Trusted Signing is Microsoft's cloud-based code signing service that:
Before starting, you'll need:
First, I need to enable the Azure Trusted Signing service in my subscription. This can be done in the Portal, or at the CLI.
# Login to Azure
az login
# Register the Microsoft.CodeSigning resource provider
az provider register --namespace Microsoft.CodeSigning
# Wait for registration to complete (takes 2-3 minutes)
az provider show --namespace Microsoft.CodeSigning --query "registrationState"
Wait until the output shows "Registered".
Now create the actual signing account. You can do this via Azure Portal or CLI.
Option A: Azure Portal (Easier for first-timers)
Option B: Azure CLI (Faster if you are a CLI person or like to drive stick shift)
# Create a resource group
az group create --name MyAppSigning --location westus2
# Create the Trusted Signing account
az trustedsigning create \
--resource-group MyAppSigning \
--account-name myapp-signing \
--location westus2 \
--sku-name Basic
Important: Note your region endpoint. Common ones are:
https://eus.codesigning.azure.net/
https://wus2.codesigning.azure.net/
I totally flaked on this and messed around for 10 min before I realized that this URL matters and is specific to your account. Remember this endpoint.
This is the most important step. Microsoft needs to verify you're a real person/organization.
Approval Time:
You'll receive an email when approved. You cannot sign any code until this is approved.
Once your identity is validated, create a certificate profile. This is what actually issues the signing certificates.
Important: Only "Public Trust" profiles prevent SmartScreen warnings. "Private Trust" is for internal apps only. This took me a second to realize also as it's not an intuitive name.
# List your Trusted Signing accounts
az trustedsigning show \
--resource-group MyAppSigning \
--account-name myapp-signing
# Should show status: "Succeeded"
Write down these values - you'll need them later:
myapp-signing
MyAppProfile
https://wus2.codesigning.azure.net/ (or your region)
MyAppSigningNow let's sign an executable on your my machine. You don't NEED to do this, but I wanted to try it locally to avoid a bunch of CI/CD runs, and I wanted to right-click the EXE and see the cert in Properties before I took it all to the cloud. The nice part about this was that I didn't need to mess with any certificates.
You need permission to actually use the signing service.
Option A: Azure Portal
Option B: Azure CLI
# Get your user object ID
$userId = az ad signed-in-user show --query id -o tsv
# Assign the role
az role assignment create \
--role "Trusted Signing Certificate Profile Signer" \
--assignee-object-id $userId \
--scope /subscriptions/YOUR_SUBSCRIPTION_ID/resourceGroups/MyAppSigning/providers/Microsoft.CodeSigning/codeSigningAccounts/myapp-signing
Replace YOUR_SUBSCRIPTION_ID with your actual subscription ID.
This is crucial - you need to login with the specific codesigning scope.
# Logout first to clear old tokens
az logout
# Login with codesigning scope
az login --use-device-code --scope "https://codesigning.azure.net/.default"
This will give you a code to enter at https://microsoft.com/devicelogin. Follow the prompts.
Why device code flow? Because Azure CLI's default authentication can conflict with Visual Studio credentials in my experience. Device code flow is more reliable for code signing.
Option A: Install Globally (Recommended for regular use)
# Install as a global tool (available everywhere)
dotnet tool install --global --prerelease sign
# Verify installation
sign --version
Option B: Install Locally (Project-specific)
# Install to current directory
dotnet tool install --tool-path . --prerelease sign
# Use with .\sign.exe
Which should I use?
Note again that code signing URL is specific to you. The tscp is your Trusted Signing Certificate Profile name and the tsa is your Trusted Signing Account name. I set *.exe to sign all the EXEs in the folder and note that the -b base directory is an absolute path, not a relative one. For me it was d:\github\WindowsEdgeLight\publish, and your mileage will vary.
# Navigate to your project folder
cd C:\MyProject
# Sign the executable
.\sign.exe code trusted-signing `
-b "C:\MyProject\publish" `
-tse "https://wus2.codesigning.azure.net" `
-tscp "MyAppProfile" `
-tsa "myapp-signing" `
*.exe `
-v Information
Parameters explained:
-b: Base directory containing files to sign
-tse: Trusted Signing endpoint (your region)
-tscp: Certificate profile name
-tsa: Trusted Signing account name
*.exe: Pattern to match files to sign
-v: Verbosity level (Trace, Information, Warning, Error)Expected output:
info: Signing WindowsEdgeLight.exe succeeded.
Completed in 2743 ms.
You can do this in PowerShell:
# Check the signature
Get-AuthenticodeSignature ".\publish\MyApp.exe" | Format-List
# Look for:
# Status: Valid
# SignerCertificate: CN=Your Name, O=Your Name, ...
# TimeStamperCertificate: Should be present
Right-click the EXE → Properties → Digital Signatures tab:
I hit all of these lol
Issue: "Please run 'az login' to set up account"
az logout then az login --use-device-code --scope "https://codesigning.azure.net/.default"Issue: "403 Forbidden"
Issue: "User account does not exist in tenant"
This is where the magic happens. I want to automatically sign every release. I'm using GitVersion so I just need to tag a commit and GitHub Actions will kick off a run. You can go look at a real run in detail at https://github.com/shanselman/WindowsEdgeLight/actions/runs/19775054123
GitHub Actions needs its own identity to sign code. We'll create a service principal (like a robot account). This is VERY different than your local signing setup.
Important: You need Owner or User Access Administrator role on your subscription to do this. If you don't have it, ask your Azure admin or a friend.
# Create service principal with signing permissions
az ad sp create-for-rbac \
--name "MyAppGitHubActions" \
--role "Trusted Signing Certificate Profile Signer" \
--scopes /subscriptions/YOUR_SUBSCRIPTION_ID/resourceGroups/MyAppSigning/providers/Microsoft.CodeSigning/codeSigningAccounts/myapp-signing \
--json-auth
This outputs JSON like this:
{
"clientId": "12345678-1234-1234-1234-123456789abc",
"clientSecret": "super-secret-value-abc123",
"tenantId": "87654321-4321-4321-4321-cba987654321",
"subscriptionId": "abcdef12-3456-7890-abcd-ef1234567890"
}
SAVE THESE VALUES IMMEDIATELY! You can't retrieve the clientSecret again. This is super important.
Alternative: Azure Portal Method
If CLI doesn't work:
AZURE_CLIENT_ID
AZURE_TENANT_ID
AZURE_CLIENT_SECRET
Go to your GitHub repository:
AZURE_CLIENT_ID - From service principal output or App registration AZURE_CLIENT_SECRET - From service principal output or Certificates & secrets AZURE_TENANT_ID - From service principal output or App registration AZURE_SUBSCRIPTION_ID - Azure Portal → Subscriptions Security Note: These secrets are encrypted and never visible in logs. Only your workflow can access them. You'll never see them again.
This is a little confusing as it's YAML, which is Satan's markup, but it's what we have sunk to as a society.
Note the dotnet-version below. Yours might be 8 or 9, etc. Also, I am building both x64 and ARM versions and I am using GitVersion so if you want a more complete build.yml, you can go here https://github.com/shanselman/WindowsEdgeLight/blob/master/.github/workflows/build.yml I am also zipping mine up and prepping my releases so my loose EXE lives in a ZIP file.
Add signing steps to your .github/workflows/build.yml:
name: Build and Sign
on:
push:
tags:
- 'v*'
workflow_dispatch:
permissions:
contents: write
jobs:
build:
runs-on: windows-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Setup .NET
uses: actions/setup-dotnet@v4
with:
dotnet-version: '10.0.x'
- name: Restore dependencies
run: dotnet restore MyApp/MyApp.csproj
- name: Build
run: |
dotnet publish MyApp/MyApp.csproj `
-c Release `
-r win-x64 `
--self-contained
# === SIGNING STEPS START HERE ===
- name: Azure Login
uses: azure/login@v2
with:
creds: '{"clientId":"${{ secrets.AZURE_CLIENT_ID }}","clientSecret":"${{ secrets.AZURE_CLIENT_SECRET }}","subscriptionId":"${{ secrets.AZURE_SUBSCRIPTION_ID }}","tenantId":"${{ secrets.AZURE_TENANT_ID }}"}'
- name: Sign executables with Trusted Signing
uses: azure/trusted-signing-action@v0
with:
azure-tenant-id: ${{ secrets.AZURE_TENANT_ID }}
azure-client-id: ${{ secrets.AZURE_CLIENT_ID }}
azure-client-secret: ${{ secrets.AZURE_CLIENT_SECRET }}
endpoint: https://wus2.codesigning.azure.net/
trusted-signing-account-name: myapp-signing
certificate-profile-name: MyAppProfile
files-folder: ${{ github.workspace }}\MyApp\bin\Release\net10.0-windows\win-x64\publish
files-folder-filter: exe
files-folder-recurse: true
file-digest: SHA256
timestamp-rfc3161: http://timestamp.acs.microsoft.com
timestamp-digest: SHA256
# === SIGNING STEPS END HERE ===
- name: Create Release
if: startsWith(github.ref, 'refs/tags/')
uses: softprops/action-gh-release@v2
with:
files: MyApp/bin/Release/net10.0-windows/win-x64/publish/MyApp.exe
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
Key points:
endpoint: Use YOUR region's endpoint (wus2, eus, etc.)
trusted-signing-account-name: Your account name (exact, case-sensitive)
certificate-profile-name: Your certificate profile name (exact, case-sensitive)
files-folder: Path to your compiled executables
files-folder-filter: File types to sign (exe, dll, etc.)
files-folder-recurse: Sign files in subfoldersNow trigger the workflow. You have two options:
Option A: Manual Trigger (Safest for testing)
Since the workflow includes workflow_dispatch:, you can trigger it manually without creating a tag:
# Trigger manually via GitHub CLI
gh workflow run build.yml
# Or go to GitHub web UI:
# Actions tab → "Build and Sign" workflow → "Run workflow" button
This is ideal for testing because:
Option B: Create a Tag (For actual releases)
# Make sure you're on your main branch with no uncommitted changes
git status
# Create and push a tag
git tag v1.0.0
git push origin v1.0.0
Use this when you're ready to create an actual release with signed binaries. This is what I am doing on my side.
Watch the progress with GitHub CLI:
# See latest runs
gh run list --limit 5
# Watch a specific run
gh run watch
# View detailed status
gh run view --log
Or visit: https://github.com/YOUR_USERNAME/YOUR_REPO/actions
Look for:
I hit a few of these, natch.
Issue: "403 Forbidden" during signing
Issue: "No files matched the pattern"
files-folder path or build artifacts in wrong location
- run: Get-ChildItem -Recurse
files-folder to matchIssue: Secrets not working
Issue: "DefaultAzureCredential authentication failed"
Azure Trusted Signing uses short-lived certificates (typically 3 days). This freaked me out but they say this is actually a security feature:
But won't my signature break after 3 days?
No, it seems that's what timestamping is for. When you sign a file:
That's why both local and GitHub Actions signing include:
timestamp-rfc3161: http://timestamp.acs.microsoft.com
Your signed executable has a certificate with:
# Using PowerShell
Get-AuthenticodeSignature "MyApp.exe" | Select-Object -ExpandProperty SignerCertificate | Format-List
# Using Windows UI
# Right-click EXE → Properties → Digital Signatures tab → Details → View Certificate
This whole thing took me about an hour to 75 minutes. It was detailed, but not deeply difficult. Misspellings, case-sensitivity, and a few account issues with Role-Based Access Control did slow me down. Hope this helps!
Written in November 2025 based on real-world implementation for WindowsEdgeLight. Your setup might vary slightly depending on Azure region and account type. Things change, be stoic.
I use my webcam constantly for streaming and I'm pretty familiar with all the internals and how the camera model on Windows works. I also use OBS extensively, so I regularly use the OBS virtual camera and flow everything through Open Broadcasting Studio.
For my podcast, I use Zencastr which is a web-based app that talks to the webcam via the browser APIs. For YouTubes, I'll use Riverside or StreamYard, also webapps.
I've done this reliably for the last several years without any trouble. Yesterday, I started seeing the most weird thing and it was absolutely perplexing and almost destroyed the day. I started seeing regular pauses in my webcam stream but only in two instances.
Micah initially said USB but my usb bus and hubs have worked reliably for years. Thought something might have changed in my El Gato capture device, but that has also been rock solid for 1/2 a decade. Then I started exploring virtual cameras and looked in the windows camera dialog under settings for a list of all virtual cameras.
Interestingly, virtual cameras don't get listed under Cameras in Settings in Windows:
From what I can tell, there's no user interface to list out all of your cameras - virtual or otherwise - in windows.
Here's a quick PowerShell script you can run to list out anything 'connected' that also includes the string "cam" in your local devices
Get-CimInstance -Namespace root\cimv2 -ClassName Win32_PnPEntity |
Where-Object { $_.Name -match 'Cam' } |
Select-Object Name, Manufacturer, PNPDeviceID
and my output
Name Manufacturer PNPDeviceID
---- ------------ -----------
Cam Link 4K Microsoft USB\VID_0FD9&PID_0066&MI_00\7&3768531A&0&0000
Digital Audio Interface (2- Cam Link 4K) Microsoft SWD\MMDEVAPI\{0.0.1.00000000}.{AF1690B6-CA2A-4AD3-AAFD-8DDEBB83DD4A}
Logitech StreamCam WinUSB Logitech USB\VID_046D&PID_0893&MI_04\7&E36D0CF&0&0004
Logitech StreamCam (Generic USB Audio) USB\VID_046D&PID_0893&MI_02\7&E36D0CF&0&0002
Logitech StreamCam Logitech USB\VID_046D&PID_0893&MI_00\7&E36D0CF&0&0000
Remote Desktop Camera Bus Microsoft UMB\UMB\1&841921D&0&RDCAMERA_BUS
Cam Link 4K (Generic USB Audio) USB\VID_0FD9&PID_0066&MI_03\7&3768531A&0&0003
Windows Virtual Camera Device Microsoft SWD\VCAMDEVAPI\B486E21F1D4BC97087EA831093E840AD2177E046699EFBF62B27304F5CCAEF57
However, when I list out my cameras using JavaScript enumerateDevices() like this
// Put variables in global scope to make them available to the browser console.
async function listWebcams() {
try {
const devices = await navigator.mediaDevices.enumerateDevices();
const webcams = devices.filter(device => device.kind === 'videoinput');
if (webcams.length > 0) {
console.log("Connected webcams:");
webcams.forEach((webcam, index) => {
console.log(`${index + 1}. ${webcam.label || `Camera ${index + 1}`}`);
});
} else {
console.log("No webcams found.");
}
} catch (error) {
console.error("Error accessing media devices:", error);
}
}
listWebcams();
I would get:
Connected webcams: test.html:11 1. Logitech StreamCam (046d:0893) test.html:11 2. OBS Virtual Camera (Windows Virtual Camera) test.html:11 3. Cam Link 4K (0fd9:0066) test.html:11 4. LSVCam test.html:11 5. OBS Virtual Camera
So, what, what's LSVCam? And depending on how I'd call it I'd get the pause and
getUserMedia error: NotReadableError NotReadableError: Could not start video source
Some apps could see this LSVCam and others couldn't. OBS really dislikes it, browsers really dislike it and it seemed to HANG on enumeration of cameras. Why can parts of Windows see this camera and others can't?
I don't know. Do you?
Regardless, it turns that it appears once in my registry, here (this is a dump of the key, you just care about the Registry PATH)
Windows Registry Editor Version 5.00
[HKEY_CLASSES_ROOT\CLSID\{860BB310-5D01-11d0-BD3B-00A0C911CE86}\Instance\LSVCam]
"FriendlyName"="LSVCam"
"CLSID"="{BA80C4AD-8AED-4A61-B434-481D46216E45}"
"FilterData"=hex:02,00,00,00,00,00,20,00,01,00,00,00,00,00,00,00,30,70,69,33,\
08,00,00,00,00,00,00,00,01,00,00,00,00,00,00,00,00,00,00,00,30,74,79,33,00,\
00,00,00,38,00,00,00,48,00,00,00,76,69,64,73,00,00,10,00,80,00,00,aa,00,38,\
9b,71,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00
If you want to get rid of it, delete HKEY_CLASSES_ROOT\CLSID\{860BB310-5D01-11d0-BD3B-00A0C911CE86}\Instance\LSVCam
WARNING: DO NOT delete the \Instance, just the LSVCam and below. I am a random person on the internet and you got here by googling, so if you mess up your machine by going into RegEdit.exe, I'm sorry to this man, but it's above me now.
Where did LSVCam.dll come from, you may ask? TikTok Live Studio, baby. Live Studio Video/Virtual Cam, I am guessing.
Directory of C:\Program Files\TikTok LIVE Studio\0.67.2\resources\app\electron\sdk\lib\MediaSDK_V1
09/18/2024 09:20 PM 218,984 LSVCam.dll
1 File(s) 218,984 bytes
This is a regression that started recently for me, so it's my opinion that they are installing a virtual camera for their game streaming feature but they are doing it poorly. It's either not completely installed, or hangs on enumeration, but the result is you'll see hangs on camera enumeration in your apps, especually browser apps that poll for cameras changes or check on a timer.
Nothing bad will happen if you delete the registry key BUT it'll show back up when you run TikTok Studio again. I still stream to TikTok, I just delete this key each time until someone on the TikTok Studio development team sees this blog post.
Hope this helps!
See the canonical version of this blog post at the Microsoft Open Source Blog!
Ten years ago, Microsoft released the source for MS-DOS 1.25 and 2.0 to the Computer History Museum, and then later republished them for reference purposes. This code holds an important place in history and is a fascinating read of an operating system that was written entirely in 8086 assembly code nearly 45 years ago.
Today, in partnership with IBM and in the spirit of open innovation, we're releasing the source code to MS-DOS 4.00 under the MIT license. There's a somewhat complex and fascinating history behind the 4.0 versions of DOS, as Microsoft partnered with IBM for portions of the code but also created a branch of DOS called Multitasking DOS that did not see a wide release.
https://github.com/microsoft/MS-DOS
A young English researcher named Connor "Starfrost" Hyde recently corresponded with former Microsoft Chief Technical Officer Ray Ozzie about some of the software in his collection. Amongst the floppies, Ray found unreleased beta binaries of DOS 4.0 that he was sent while he was at Lotus. Starfrost reached out to the Microsoft Open Source Programs Office (OSPO) to explore releasing DOS 4 source, as he is working on documenting the relationship between DOS 4, MT-DOS, and what would eventually become OS/2. Some later versions of these Multitasking DOS binaries can be found around the internet, but these new Ozzie beta binaries appear to be much earlier, unreleased, and also include the ibmbio.com source.
Scott Hanselman, with the help of internet archivist and enthusiast Jeff Sponaugle, has imaged these original disks and carefully scanned the original printed documents from this "Ozzie Drop". Microsoft, along with our friends at IBM, think this is a fascinating piece of operating system history worth sharing.
Jeff Wilcox and OSPO went to the Microsoft Archives, and while they were unable to find the full source code for MT-DOS, they did find MS DOS 4.00, which we're releasing today, alongside these additional beta binaries, PDFs of the documentation, and disk images. We will continue to explore the archives and may update this release if more is discovered.
Thank you to Ray Ozzie, Starfrost, Jeff Sponaugle, Larry Osterman, our friends at the IBM OSPO, as well as the makers of such digital archeology software including, but not limited to Greaseweazle, Fluxengine, Aaru Data Preservation Suite, and the HxC Floppy Emulator. Above all, thank you to the original authors of this code, some of whom still work at Microsoft and IBM today!
If you'd like to run this software yourself and explore, we have successfully run it directly on an original IBM PC XT, a newer Pentium, and within the open source PCem and 86box emulators.
I've been doing not just Unit Testing for my sites but full on Integration Testing and Browser Automation Testing as early as 2007 with Selenium. Lately, however, I've been using the faster and generally more compatible Playwright. It has one API and can test on Windows, Linux, Mac, locally, in a container (headless), in my CI/CD pipeline, on Azure DevOps, or in GitHub Actions.
For me, it's that last moment of truth to make sure that the site runs completely from end to end.
I can write those Playwright tests in something like TypeScript, and I could launch them with node, but I like running end unit tests and using that test runner and test harness as my jumping off point for my .NET applications. I'm used to right clicking and "run unit tests" or even better, right click and "debug unit tests" in Visual Studio or VS Code. This gets me the benefit of all of the assertions of a full unit testing framework, and all the benefits of using something like Playwright to automate my browser.
In 2018 I was using WebApplicationFactory and some tricky hacks to basically spin up ASP.NET within .NET (at the time) Core 2.1 within the unit tests and then launching Selenium. This was kind of janky and would require to manually start a separate process and manage its life cycle. However, I kept on with this hack for a number of years basically trying to get the Kestrel Web Server to spin up inside of my unit tests.
I've recently upgraded my main site and podcast site to .NET 8. Keep in mind that I've been moving my websites forward from early early versions of .NET to the most recent versions. The blog is happily running on Linux in a container on .NET 8, but its original code started in 2002 on .NET 1.1.
Now that I'm on .NET 8, I scandalously discovered (as my unit tests stopped working) that the rest of the world had moved from IWebHostBuilder to IHostBuilder five version of .NET ago. Gulp. Say what you will, but the backward compatibility is impressive.
As such my code for Program.cs changed from this
public static void Main(string[] args)
{
CreateWebHostBuilder(args).Build().Run();
}
public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
WebHost.CreateDefaultBuilder(args)
.UseStartup<Startup>();
to this:
public static void Main(string[] args)
{
CreateHostBuilder(args).Build().Run();
}
public static IHostBuilder CreateHostBuilder(string[] args) =>
Host.CreateDefaultBuilder(args).
ConfigureWebHostDefaults(WebHostBuilder => WebHostBuilder.UseStartup<Startup>());
Not a major change on the outside but tidies things up on the inside and sets me up with a more flexible generic host for my web app.
My unit tests stopped working because my Kestral Web Server hack was no longer firing up my server.
Here is an example of my goal from a Playwright perspective within a .NET NUnit test.
[Test]
public async Task DoesSearchWork()
{
await Page.GotoAsync(Url);
await Page.Locator("#topbar").GetByRole(AriaRole.Link, new() { Name = "episodes" }).ClickAsync();
await Page.GetByPlaceholder("search and filter").ClickAsync();
await Page.GetByPlaceholder("search and filter").TypeAsync("wife");
const string visibleCards = ".showCard:visible";
var waiting = await Page.WaitForSelectorAsync(visibleCards, new PageWaitForSelectorOptions() { Timeout = 500 });
await Expect(Page.Locator(visibleCards).First).ToBeVisibleAsync();
await Expect(Page.Locator(visibleCards)).ToHaveCountAsync(5);
}
I love this. Nice and clean. Certainly here we are assuming that we have a URL in that first line, which will be localhost something, and then we assume that our web application has started up on its own.
Here is the setup code that starts my new "web application test builder factory," yeah, the name is stupid but it's descriptive. Note the OneTimeSetUp and the OneTimeTearDown. This starts my web app within the context of my TestHost. Note the :0 makes the app find a port which I then, sadly, have to dig out and put into the Url private for use within my Unit Tests. Note that the <Startup> is in fact my Startup class within Startup.cs which hosts my app's pipeline and Configure and ConfigureServices get setup here so routing all works.
private string Url;
private WebApplication? _app = null;
[OneTimeSetUp]
public void Setup()
{
var builder = WebApplicationTestBuilderFactory.CreateBuilder<Startup>();
var startup = new Startup(builder.Environment);
builder.WebHost.ConfigureKestrel(o => o.Listen(IPAddress.Loopback, 0));
startup.ConfigureServices(builder.Services);
_app = builder.Build();
// listen on any local port (hence the 0)
startup.Configure(_app, _app.Configuration);
_app.Start();
//you are kidding me
Url = _app.Services.GetRequiredService<IServer>().Features.GetRequiredFeature<IServerAddressesFeature>().Addresses.Last();
}
[OneTimeTearDown]
public async Task TearDown()
{
await _app.DisposeAsync();
}
So what horrors are buried in WebApplicationTestBuilderFactory? The first bit is bad and we should fix it for .NET 9. The rest is actually every nice, with a hat tip to David Fowler for his help and guidance! This is the magic and the ick in one small helper class.
public class WebApplicationTestBuilderFactory
{
public static WebApplicationBuilder CreateBuilder<T>() where T : class
{
//This ungodly code requires an unused reference to the MvcTesting package that hooks up
// MSBuild to create the manifest file that is read here.
var testLocation = Path.Combine(AppContext.BaseDirectory, "MvcTestingAppManifest.json");
var json = JsonObject.Parse(File.ReadAllText(testLocation));
var asmFullName = typeof(T).Assembly.FullName ?? throw new InvalidOperationException("Assembly Full Name is null");
var contentRootPath = json?[asmFullName]?.GetValue<string>();
//spin up a real live web application inside TestHost.exe
var builder = WebApplication.CreateBuilder(
new WebApplicationOptions()
{
ContentRootPath = contentRootPath,
ApplicationName = asmFullName
});
return builder;
}
}
The first 4 lines are nasty. Because the test runs in the context of a different directory and my website needs to run within the context of its own content root path, I have to force the content root path to be correct and the only way to do that is by getting the apps base directory from a file generated within MSBuild from the (aging) MvcTesting package. The package is not used, but by referencing it it gets into the build and makes that file that I then use to pull out the directory.
If we can get rid of that "hack" and pull the directory from context elsewhere, then this helper function turns into a single line and .NET 9 gets WAY WAY more testable!
Now I can run my Unit Tests AND Playwright Browser Integration Tests across all OS's, headed or headless, in docker or on the metal. The site is updated to .NET 8 and all is right with my code. Well, it runs at least. ;)
There are many let's encrypt automatic tools for azure but I also wanted to see if I could use certbot in wsl to generate a wildcard certificate for the azure Friday website and then upload the resulting certificates to azure app service.
Azure app service ultimately needs a specific format called dot PFX that includes the full certificate path and all intermediates.
Per the docs, App Service private certificates must meet the following requirements:
If you have a PFX that doesn't meet all these requirements you can have Windows reencrypt the file.
I use WSL and certbot to create the cert, then I import/export in Windows and upload the resulting PFX.
Within WSL, install certbot:
sudo apt update
sudo apt install python3 python3-venv libaugeas0
sudo python3 -m venv /opt/certbot/
sudo /opt/certbot/bin/pip install --upgrade pip
sudo /opt/certbot/bin/pip install certbot
Then I generate the cert. You'll get a nice text UI from certbot and update your DNS as a verification challenge. Change this to make sure it's two lines, and your domains and subdomains are correct and your paths are correct.
sudo certbot certonly --manual --preferred-challenges=dns --email YOUR@EMAIL.COM
--server https://acme-v02.api.letsencrypt.org/directory
--agree-tos --manual-public-ip-logging-ok -d "azurefriday.com" -d "*.azurefriday.com"
sudo openssl pkcs12 -export -out AzureFriday2023.pfx
-inkey /etc/letsencrypt/live/azurefriday.com/privkey.pem
-in /etc/letsencrypt/live/azurefriday.com/fullchain.pem
I then copy the resulting file to my desktop (check your desktop path) so it's now in the Windows world.
sudo cp AzureFriday2023.pfx /mnt/c/Users/Scott/OneDrive/Desktop
Now from Windows, import the PFX, note the thumbprint and export that cert.
Import-PfxCertificate -FilePath "AzureFriday2023.pfx" -CertStoreLocation Cert:\LocalMachine\My
-Password (ConvertTo-SecureString -String 'PASSWORDHERE' -AsPlainText -Force) -Exportable
Export-PfxCertificate -Cert Microsoft.PowerShell.Security\Certificate::LocalMachine\My\597THISISTHETHUMBNAILCF1157B8CEBB7CA1
-FilePath 'AzureFriday2023-fixed.pfx' -Password (ConvertTo-SecureString -String 'PASSWORDHERE' -AsPlainText -Force)
Then upload the cert to the Certificates section of your App Service, under Bring Your Own Cert.
Then under Custom Domains, click Update Binding and select the new cert (with the latest expiration date).
Next step is to make this even more automatic or select a more automated solution but for now, I'll worry about this in September and it solved my expensive Wildcard Domain issue.
GitHub Next has this cool project that is basically Copilot for the CLI (command line interface). You can sign up for their waitlist at the Copilot for CLI site.
Copilot for CLI provides three shell commands:
??,git?andgh?
This is cool and all, but I use PowerShell. Turns out these ?? commands are just router commands to a larger EXE called github-copilot-cli. So if you go "?? something" you're really going "github-copilot-cli what-the-shell something."
So this means I should be able to to do the same/similar aliases for my PowerShell prompt AND change the injected prompt (look at me I'm a prompt engineer) to add 'use powershell to.'
Now it's not perfect, but hopefully it will make the point to the Copilot CLI team that PowerShell needs love also.
Here are my aliases. Feel free to suggest if these suck. Note the addition of "user powershell to" for the ?? one. I may make a ?? and a p? where one does bash and one does PowerShell. I could also have it use wsl.exe and shell out to bash. Lots of possibilities.
function ?? {
$TmpFile = New-TemporaryFile
github-copilot-cli what-the-shell ('use powershell to ' + $args) --shellout $TmpFile
if ([System.IO.File]::Exists($TmpFile)) {
$TmpFileContents = Get-Content $TmpFile
if ($TmpFileContents -ne $nill) {
Invoke-Expression $TmpFileContents
Remove-Item $TmpFile
}
}
}
function git? {
$TmpFile = New-TemporaryFile
github-copilot-cli git-assist $args --shellout $TmpFile
if ([System.IO.File]::Exists($TmpFile)) {
$TmpFileContents = Get-Content $TmpFile
if ($TmpFileContents -ne $nill) {
Invoke-Expression $TmpFileContents
Remove-Item $TmpFile
}
}
}
function gh? {
$TmpFile = New-TemporaryFile
github-copilot-cli gh-assist $args --shellout $TmpFile
if ([System.IO.File]::Exists($TmpFile)) {
$TmpFileContents = Get-Content $TmpFile
if ($TmpFileContents -ne $nill) {
Invoke-Expression $TmpFileContents
Remove-Item $TmpFile
}
}
}
It also then offers to run the command. Very smooth.
Hope you like it. Lots of fun stuff happening in this space.
Mastodon is a free, open-source social networking service that is decentralized and distributed. It was created in 2016 as an alternative to centralized social media platforms such as Twitter and Facebook.
One of the key features of Mastodon is the use of the WebFinger protocol, which allows users to discover and access information about other users on the Mastodon network. WebFinger is a simple HTTP-based protocol that enables a user to discover information about other users or resources on the internet by using their email address or other identifying information. The WebFinger protocol is important for Mastodon because it enables users to find and follow each other on the network, regardless of where they are hosted.
WebFinger uses a "well known" path structure when calling an domain. You may be familiar with the robots.txt convention. We all just agree that robots.txt will sit at the top path of everyone's domain.
The WebFinger protocol is a simple HTTP-based protocol that enables a user or search to discover information about other users or resources on the internet by using their email address or other identifying information. My is first name at last name .com, so...my personal WebFinger API endpoint is here https://www.hanselman.com/.well-known/webfinger
The idea is that...
A user sends a WebFinger request to a server, using the email address or other identifying information of the user or resource they are trying to discover.
The server looks up the requested information in its database and returns a JSON object containing the information about the user or resource. This JSON object is called a "resource descriptor."
The user's client receives the resource descriptor and displays the information to the user.
The resource descriptor contains various types of information about the user or resource, such as their name, profile picture, and links to their social media accounts or other online resources. It can also include other types of information, such as the user's public key, which can be used to establish a secure connection with the user.
There's a great explainer here as well. From that page:
When someone searches for you on Mastodon, your server will be queried for accounts using an endpoint that looks like this:
GET https://${MASTODON_DOMAIN}/.well-known/webfinger?resource=acct:${MASTODON_USER}@${MASTODON_DOMAIN}
Note that Mastodon user names start with @ so they are @username@someserver.com. Just like twiter would be @shanselman@twitter.com I can be @shanselman@hanselman.com now!
So perhaps https://www.hanselman.com/.well-known/webfinger?resource=acct:FRED@HANSELMAN.COM
Mine returns
{
"subject":"acct:shanselman@hachyderm.io",
"aliases":
[
"https://hachyderm.io/@shanselman",
"https://hachyderm.io/users/shanselman"
],
"links":
[
{
"rel":"http://webfinger.net/rel/profile-page",
"type":"text/html",
"href":"https://hachyderm.io/@shanselman"
},
{
"rel":"self",
"type":"application/activity+json",
"href":"https://hachyderm.io/users/shanselman"
},
{
"rel":"http://ostatus.org/schema/1.0/subscribe",
"template":"https://hachyderm.io/authorize_interaction?uri={uri}"
}
]
}
This file should be returned as a mime type of application/jrd+json
My site is an ASP.NET Razor Pages site, so I just did this in Startup.cs to map that well known URL to a page/route that returns the JSON needed.
services.AddRazorPages().AddRazorPagesOptions(options =>
{
options.Conventions.AddPageRoute("/robotstxt", "/Robots.Txt"); //i did this before, not needed
options.Conventions.AddPageRoute("/webfinger", "/.well-known/webfinger");
options.Conventions.AddPageRoute("/webfinger", "/.well-known/webfinger/{val?}");
});
then I made a webfinger.cshtml like this. Note I have to double escape the @@ sites because it's Razor.
@page
@{
Layout = null;
this.Response.ContentType = "application/jrd+json";
}
{
"subject":"acct:shanselman@hachyderm.io",
"aliases":
[
"https://hachyderm.io/@@shanselman",
"https://hachyderm.io/users/shanselman"
],
"links":
[
{
"rel":"http://webfinger.net/rel/profile-page",
"type":"text/html",
"href":"https://hachyderm.io/@@shanselman"
},
{
"rel":"self",
"type":"application/activity+json",
"href":"https://hachyderm.io/users/shanselman"
},
{
"rel":"http://ostatus.org/schema/1.0/subscribe",
"template":"https://hachyderm.io/authorize_interaction?uri={uri}"
}
]
}
This is a static response, but if I was hosting pages for more than one person I'd want to take in the url with the user's name, and then map it to their aliases and return those correctly.
Even easier, you can just use the JSON file of your own Mastodon server's webfinger response and SAVE IT as a static json file and copy it to your own server!
As long as your server returns the right JSON from that well known URL then it'll work.
So this is my template https://hachyderm.io/.well-known/webfinger?resource=acct:shanselman@hachyderm.io from where I'm hosted now.
If you want to get started with Mastodon, start here. https://github.com/joyeusenoelle/GuideToMastodon/ it feels like Twitter circa 2007 except it's not owned by anyone and is based on web standards like ActivityPub.
Hope this helps!
I have been blogging here for the last 20 years. Every Tuesday and Thursday, quite consistently, for two decades. But last year, without planning it, I got tired and stopped. Not sure why. It didn't correspond with any life events. Nothing interesting or notable happened. I just stopped.
I did find joy on TikTok and amassed a small group of like-minded followers there. I enjoy my YouTube as well, and my weekly podcast is going strong with nearly 900 (!) episodes of interviews with cool people. I've also recently started posting on Mastodon (a fediverse (federated universe)) Twitter alternative that uses the ActivityPub web standard. I see that Mark Downie has been looking at ActivityPub as well for DasBlog (the blog engine that powers this blog) so I need to spend sometime with Mark soon.
Being consistent is a hard thing, and I think I did a good job. I gave many talks over many years about Personal Productivity but I always mentioned doing what "feeds your spirit." For a minute here the blog took a backseat, and that's OK. I filled that (spare) time with family time, personal projects, writing more code, 3d printing, games, taekwondo, and a ton of other things.
Going forward I will continue to write and share across a number of platforms, but it will continue to start here as it's super important to Own Your Words. Keep taking snapshots and backups of your keystrokes as you never know when your chosen platform might change or go away entirely.
I'm still here. I hope you are too! I will see you soon.
I am not a Home Assistant expert, but it's clearly a massive and powerful ecosystem. I've interviewed the creator of Home Assistant on my podcast and I encourage you to check out that chat.
Home Assistant can quickly become a hobby that overwhelms you. Every object (entity) in your house that is even remotely connected can become programmable. Everything. Even people! You can declare that any name:value pair that (for example) your phone can expose can be consumable by Home Assistant. Questions like "is Scott home" or "what's Scott's phone battery" can be associated with Scott the Entity in the Home Assistant Dashboard.
I was amazed at the devices/objects that Home Assistant discovered that it could automate. Lights, remotes, Spotify, and more. You'll find that any internally connected device you have likely has an Integration available.
Temperature, Light Status, sure, that's easy Home Automation. But integrations and 3rd party code can give you details like "Is the Living Room dark" or "is there motion in the driveway." From these building blocks, you can then build your own IFTTT (If This Then That) automations, combining not just two systems, but any and all disparate systems.
What's the best part? This all runs LOCALLY. Not in a cloud or the cloud or anyone's cloud. I've got my stuff running on a Raspberry Pi 4. Even better I put a Power Over Ethernet (PoE) hat on my Rpi so I have just one network wire into my hub that powers the Pi.
I believe setting up Home Assistant on a Pi is the best and easiest way to get started. That said, you can also run in a Docker Container, on a Synology or other NAS, or just on Windows or Mac in the background. It's up to you. Optionally, you can pay Nabu Casa $5 for remote (outside your house) network access via transparent forwarding. But to be clear, it all still runs inside your house and not in the cloud.
OK, to the main point. I used to have an Amazon Ring Doorbell that would integrate with Amazon Alexa and when you pressed the doorbell it would say "Someone is at the front door" on our all Alexas. It was a lovely little integration that worked nicely in our lives.
However, I swapped out the Ring for a Unifi Protect G4 Doorbell for a number of reasons. I don't want to pump video to outside services, so this doorbell integrates nicely with my existing Unifi installation and records video to a local hard drive. However, I lose any Alexa integration and this nice little "someone is at the door" announcement. So this seems like a perfect job for Home Assistant.
Here's the general todo list:
I recommend going into your Alexa app and making a Multi-room Speaker Group called "everywhere." Not only because it's nice to be able to say "play the music everywhere" but you can also target that "Everywhere" group in Home Assistant.
Go into your Home Assistant UI at http://homeassistant.local:8123/ and into Developer Tools. Under Services, try pasting in this YAML and clicking "call service."
service: notify.alexa_media_everywhere
data:
message: Someone is at the front door, this is a test
data:
type: announce
method: speak
If that works, you know you can automate Alexa and make it say things. Now, go to Configuration, Automation, and Add a new Automation. Here's mine. I used the UI to create it. Note that your Entity names may be different if you give your front doorbell camera a different name.
Notice the format of Data, it's name value pairs within a single field's value.
...but it also exists in a file called Automations.yaml. Note that the "to: 'on'" trigger is required or you'll get double announcements, one for each state change in the doorbell.
- id: '1640995128073'
alias: G4 Doorbell Announcement with Alexa
description: G4 Doorbell Announcement with Alexa
trigger:
- platform: state
entity_id: binary_sensor.front_door_doorbell
to: 'on'
condition: []
action:
- service: notify.alexa_media_everywhere
data:
data:
type: announce
method: speak
message: Someone is at the front door
mode: single
It works! There's a ton of cool stuff I can automate now!
Sponsor: Make login Auth0’s problem. Not yours. Provide the convenient login features your customers want, like social login, multi-factor authentication, single sign-on, passwordless, and more. Get started for free.
I was reading Gabby's blog post about the new TypeScript/JavaScript project experience in Visual Studio 2022. You should read the docs on JavaScript and TypeScript in Visual Studio 2022.
If you're used to ASP.NET apps when you think about apps that are JavaScript heavy, "front end apps" or TypeScript focused, it can be confusing as to "where does .NET fit in?"
You need to consider the responsibilities of your various projects or subsystems and the multiple totally valid ways you can build a web site or web app. Let's consider just a few:
VS2022 brings JavaScript and TypeScript support into VS with a full JavaScript Language Service based on TS. It provides a TypeScript NuGet Package so you can build your whole app with MSBuild and VS will do the right thing.
NEW: Starting in Visual Studio 2022, there is a new JavaScript/TypeScript project type (.esproj) that allows you to create standalone Angular, React, and Vue projects in Visual Studio.
The .esproj concept is great for folks familiar with Visual Studio as we know that a Solution contains one or more Projects. Visual Studio manages files for a single application in a Project. The project includes source code, resources, and configuration files. In this case we can have a .csproj for a backend Web API and an .esproj that uses a client side template like Angular, React, or Vue.
Thing is, historically when Visual Studio supported Angular, React, or Vue, it's templates were out of date and not updated enough. VS2022 uses the native CLIs for these front ends, solving that problem with Angular CLI, Create React App, and Vue CLI.
If I am in VS and go "File New Project" there are Standalone templates that solve Example 2 above. I'll pick JavaScript React.
Then I'll click "Add integration for Empty ASP.NET Web API. This will give me a frontend with javascript ready to call a ASP.NET Web API backend. I'll follow along here.
It then uses the React CLI to make the front end, which again, is cool as it's whatever version I want it to be.
Then I'll add my ASP.NET Web API backend to the same solution, so now I have an esproj and a csproj like this
Now I have a nice clean two project system - in this case more JavaScript focused than .NET focused. This one uses npm to startup the project using their web development server and proxyMiddleware to proxy localhost:3000 calls over to the ASP.NET Web API project.
Here is a React app served by npm calling over to the Weather service served from Kestrel on ASP.NET.
This is inverted than most ASP.NET Folks are used to, and that's OK. This shows me that Visual Studio 2022 can support either development style, use the CLI that is installed for whatever Frontend Framework, and allow me to choose what web server and web browser (via Launch.json) I want.
If you want to flip it, and put ASP.NET Core as the primary and then bring in some TypeScript/JavaScript, follow this tutorial because that's also possible!
Sponsor: Make login Auth0’s problem. Not yours. Provide the convenient login features your customers want, like social login, multi-factor authentication, single sign-on, passwordless, and more. Get started for free.
I've talked about how I love a nice pretty prompt in my Windows Terminal and made videos showing in detail how to do it. I've also worked with my buddy TooTallNate to put my real-time blood sugar into a bash or PowerShell prompt, but this was back in 2017.
Now that I'm "Team OhMyPosh" I have been meaning to write a Nightscout "segment" for my prompt. Nightscout is an open source self-hosted (there are commercial hosts also like T1Pal) website and API for remote display of real-time and near-real-time glucose readings for Diabetics like myself.
Since my body has an active REST API where I can just do an HTTP GET (via curl or whatever) and see my blood sugar, it clearly belongs in a place of honor, just like my current Git Branch!
Oh My Posh supports configurable "segments" and now there's a beta (still needs mmol and stale readings support) Nightscout segment that you can setup in just a few minutes!
This prompt works in ANY shell on ANY os! You can do this in zsh, PowerShell, Bash, whatever makes you happy.
Here is a YouTube of Jan from OhMyPosh and I coding the segment LIVE in Go.
If you have an existing OhMyPosh json config, you can just add another segment like this. Make sure your Nightscout URL includes a secure Token or is public (up to you). Note also that I setup "if/then" rules in my background_templates. These are optional and up to you to change to your taste. I set my background colors to red, yellow, green depending on sugar numbers. I also have a foreground template that is not really used, as you can see it always evaluates to black #000, but it shows you how you could set it to white text on a darker background if you wanted.
{
"type": "nightscout",
"style": "diamond",
"foreground": "#ffffff",
"background": "#ff0000",
"background_templates": [
"{{ if gt .Sgv 150 }}#FFFF00{{ end }}",
"{{ if lt .Sgv 60 }}#FF0000{{ end }}",
"#00FF00"
],
"foreground_templates": [
"{{ if gt .Sgv 150 }}#000000{{ end }}",
"{{ if lt .Sgv 60 }}#000000{{ end }}",
"#000000"
],
"leading_diamond": "",
"trailing_diamond": "\uE0B0",
"properties": {
"url": "https://YOURNIGHTSCOUTAPP.herokuapp.com/api/v1/entries.json?count=1&token=APITOKENFROMYOURADMIN",
"http_timeout": 1500,
"template": " {{.Sgv}}{{.TrendIcon}}"
}
},
By default we will only go out and hit your Nightscout instance every 5 min, only when the prompt is repainted, and we'll only wait 1500ms before giving up. You can set that "http_timeout" (how long before we give up) if you feel this slows you down. It'll be cached for 5 min so it's unlikely to b something you'll notice. The benefit of this new OhMyPosh segment over the previous solution is that it requires no additional services/chron jobs and can be setup extremely quickly. Note also that you can customize your template with NerdFonts. I've included a tiny syringe!
Next I'll hope to improve the segment with mmol support as well as strikeout style for "stale" (over 15 min old) results. You're also welcome to help out by watching our YouTube and submitting a PR!
Sponsor: Make login Auth0’s problem. Not yours. Provide the convenient login features your customers want, like social login, multi-factor authentication, single sign-on, passwordless, and more. Get started for free.
For the PC hardware space, the marquee announcement coming out of CES 2026 this year is without a doubt the launch of Intel’s Core Ultra Series 3 of chips. Going by the codename of Panther Lake, the latest of Intel’s lakes marks two very important milestones for the company. Besides being the launch of a […]
The post Intel Launches Core Ultra Series 3 Mobile Processors – Panther Lake Roars to Life appeared first on ServeTheHome.
Alongside AMD’s numerous client-focused hardware announcements during their CES 2026 keynote, the company also devoted a bit of attention to their data center/server products. Though the company is essentially mid-cycle on its major data center products right now – and thus is not launching anything in the immediate timeframe – AMD still opted to use […]
The post AMD’s EPYC Venice, Instinct MI455X, & Helios Hardware On Display for First Time at CES 2026 appeared first on ServeTheHome.
Marvell announced that it has reached a deal to acquire XConn Technologies in a push for CXL memory and PCIe switching
The post Marvell Announces XConn Technologies Acquisition in CXL and PCIe Push appeared first on ServeTheHome.
Among a spate of AMD announcements during the company’s CES 2026 opening keynote, CEO Dr. Lisa Su briefly teased a forthcoming AMD-branded AI development box, dubbed the Ryzen AI Halo. Seemingly taking a page from similar AI development boxes that have popped up over the last year – both those built using AMD’s hardware and […]
The post AMD Teases Ryzen AI Halo, a ROCm Ecosystem AI Development Mini-PC appeared first on ServeTheHome.
Alongside AMD’s slew of consumer-related product announcements with desktop and mobile Ryzen processors, the company also took a moment of its time during its CES 2026 presentation to address the embedded market. The tangentially-related cousin to the consumer market, AMD’s embedded lineup of processors are aimed at the automotive and industrial markets, as well as […]
The post AMD Announces Zen 5-based Ryzen AI Embedded P100 Family of Chips appeared first on ServeTheHome.
Kicking off AMD’s slate of consumer announcements for this year’s CES trade show, the company brought to the shows several new chip SKUs for the consumer market. Altogether the company is launching two new Ryzen AI Max+ processors for AI developers, a new desktop Ryzen 9000X3D chip for gamers, and for the mobile market a […]
The post AMD Reveals New Ryzen AI 400 Series, Ryzen AI Max+, and Ryzen 7 9850X3D Chips At CES 2026 appeared first on ServeTheHome.
Nvidia's H200 GPUs could begin trickling into China as soon as this quarter, but there's a catch. Due to all the geopolitical turmoil that's ravaged US-China trade relations over the past year, buyers may need to pay up front for the coveted AI accelerators. And they won't get a refund if China decides to block the imports!…
We hope you like more AI in your Gmail inbox, because Google is "bringing Gmail into the Gemini era." It'll be on by default, but the good news is that you can disable it. …
Cisco patched a bug in its Identity Services Engine (ISE) and ISE Passive Identity Connector (ISE-PIC) products that allows remote attackers with admin-level privileges to access sensitive information - and warned that a public, proof-of-concept exploit for the flaw exists online.…
Today, it is hard to escape LLM bots and the endless slop they emit, but the Linux kernel might be largely safe … for now.…
Datacenter building decisions tend to fall into two camps with colocation providers plumping for urban areas while hyperscalers seek sites where electricity, land, and construction costs come cheaper.…
NASA has postponed today's spacewalk outside the International Space Station (ISS) due to an undisclosed "medical concern" with a crew member.…
If 2025 was meant to be the year ransomware started dying, nobody appears to have told the attackers.…
Airlines operating in the US may have to upgrade their aircraft radio altimeters again at a cost of billions of dollars, to avoid potential interference with cell networks following the Trump administration's decision last year to auction off additional spectrum to bidders.…
CISA has added a pair of security holes to its actively exploited list, warning that attackers are now abusing a maximum-severity bug in HPE's OneView management software and a years-old flaw in Microsoft Office.…
Google has confirmed there will be two code dumps to the Android Open Source Project (AOSP) per year, down from the four developers have become accustomed to.…
Elon Musk's X platform is under fire as UK regulators close in on mounting reports that the platform's AI chatbot, Grok, is generating sexual imagery without users' consent.…
A maximum-severity bug in the popular automation platform n8n has left an estimated 100,000 servers wide open to complete takeover, courtesy of a flaw so bad it doesn't even require logging in.…
Security researchers at Radware say they've identified several vulnerabilities in OpenAI's ChatGPT service that allow the exfiltration of personal information.…
Interview With everyone from would-be developers to six-year-old kids jumping on the vibe coding bandwagon, it shouldn't be surprising that criminals like automated coding tools too.…
Scientists have developed a synthetic skin capable of mimicking some of the best camouflage skills in nature that could also have applications in soft robotics and advanced displays.…
Logitech says an expired developer certificate is to blame after swaths of customers were left infuriated when their mice malfunctioned.…
Cloudflare has poured cold water on a theory that the USA’s incursion into Venezuela coincided with a cyberattack on telecoms infrastructure.…
AMD teased its next-generation of AI accelerators at CES 2026, with CEO Lisa Su boasting the the MI500-series will deliver a 1,000x uplift in performance over its two-year-old MI300X GPUs.…
IBM describes its coding agent thus: "Bob is your AI software development partner that understands your intent, repo, and security standards." Unfortunately, Bob doesn't always follow those security standards.…
Researchers in Finland have found a new way to capture carbon dioxide from ambient air that they say is more efficient than existing methods, cheap to produce, reusable, and allows for easy recycling of captured CO₂. …
Accenture plans to buy UK-based AI firm Faculty, a Palantir competitor, and onboard the company’s CEO as Accenture’s new chief technology officer. The move suggests the two companies, while partners today, could start taking each others' business.…
WD Black and Blue SSDs are some of the most widely recognized client drives on the market, but their branding is about to disappear. Following Western Digital's flash-business spinoff, SanDisk announced it was retiring the beloved names and rebranding its NVMe lineup under the SANDISK Optimus banner.…
exclusive The European Space Agency on Wednesday confirmed yet another massive security breach, and told The Register that the data thieves responsible will be subject to a criminal investigation. And this could be a biggie.…
The US government has secured a guilty plea from a stalkerware maker in federal court, marking just the second time in more than a decade that the US has managed to prosecute a consumer spyware vendor successfully. …
The US government has announced contracts for new radar infrastructure as part of its long-running effort to replace the country's aging air traffic control system.…
With less than a month to go until NASA attempts to send astronauts around the Moon, the agency is demolishing facilities that got it there the first time around.…
Fancy having an AI system packed with Nvidia H200 GPUs that you can take with you from place to place? According to hardware maker Odinn, now you can, so long as you don't mind carrying a 77-pound (35 kg) box around.…
Microsoft has backed away from planned changes to Exchange Online after customers objected to limits designed to curb outbound email abuse.…
The US state of Virginia forfeited $1.6 billion in tax revenue through datacenter exemptions in fiscal 2025 – up 118 percent on the prior year – as the AI-driven construction boom accelerates.…
Opinion Ever since Linux got a graphical desktop, you could middle-click to paste – but if GNOME gets its way, that's going away soon, and from Firefox too.…
The Post Office's Horizon computer system may have been deployed earlier than thought, potentially affecting which convictions get automatically quashed under legislation introduced to speed up justice in one of the biggest scandals in recent British history, MPs heard yesterday.…
Updated The UK's Ministry of Justice spent £50 million ($67 million) on cybersecurity improvements at the Legal Aid Agency (LAA) before the high-profile cyberattack it disclosed last year.…
Brit luxury automaker Jaguar Land Rover has reported devastating preliminary Q3 results that lay bare the cascading consequences of a crippling cyberattack, revealing wholesale volumes collapsed more than two-fifths year-on-year.…
Bork!Bork!Bork! The baddest of AI bad guys, the Terminator, has confirmed what the vast majority of IT professionals already know. The machines are not about to rise, not until they can deal with that pesky battery voltage.…
Some HSBC mobile banking customers in the UK report being locked out of the bank's app after installing the Bitwarden password manager via an open source app catalog.…
The UK's Department for Work and Pensions (DWP) is set to introduce a conversational AI platform it hopes will steer calls from citizens with queries about their benefits. The contract is worth up to £23 million.…
If there was a kingdom of laptop screen flexibility, Lenovo would take the crown. Last year, the company released the ThinkBook Plus Gen 6, with a mechanical screen that could roll out to increase its size from 14 to 16.7 inches. Now, it’s back with the ThinkPad Rollable XD concept laptop that expands from 13.3 to 16 inches at the touch of a button or a swipe, along with the ThinkBook Plus Gen 7 Auto Twist, which uses a motor to rotate its screen and follow you around the room.…
A year after a series of fires obliterated communities in Los Angeles, Amazon's Ring security service has announced a feature called Fire Watch intended to mitigate future wildfire risk.…
Last fall, Jakub Ciolek reported two denial-of-service bugs in Argo CD, a popular Kubernetes controller, via HackerOne's Internet Bug Bounty (IBB) program. Both were assigned CVEs and have since been fixed. But instead of receiving an $8,500 reward for the two flaws, Ciolek says, HackerOne ghosted him for months.…
The data platform Snowflake is putting Google's Gemini to work inside its Cortex AI, aiming to give customers access to a foundational model within the boundaries of their data environment across supported clouds, the company told The Register.…
Brave Software has reworked its browser's Rust-based adblock engine to make it significantly more memory efficient and perhaps more secure. So you get fewer ads now with fewer MB of RAM.…
CES 2026 Remember when Elon Musk predicted that there would be thousands of Optimus robots at Tesla factories by the end of 2025? Well, that didn't happen, but competitor Boston Dynamics has just announced that its humanoid robot, Atlas, is going to the big time.…
Internet service provider Brightspeed confirmed that it's investigating criminals' claims that they stole more than a million customers' records and have listed them for sale for three bitcoin, or about $276,370. …
Not even Legos are safe from the inexorable march of smart technology, as the Danish construction toy stalwart introduced a new tech-in-a-brick Smart Play system at CES this week. …
What if, rather than make a Linux distro that can run Windows apps, you built the whole distro around Windows binaries instead?…
For many, the start of a new year is a time to take stock. For Microsoft, it was a time to stop giving it as the company kicked off 2026 with a bug that broke Excel's StockHistory function.…
Microsoft has bought Osmos, an AI-assisted data engineering platform, in a bid to enrich its Fabric data platform, encroaching on so-called partners' markets.…
Memory prices are set to spike again as chipmakers prioritize AI server production over consumer devices, with analysts warning of a high double-digit jump in Q1 2026 alone as demand outpaces supply.…
Russia-linked hackers are sneaking malware into European hotels and other hospitality outfits by tricking staff into installing it themselves through fake Windows Blue Screen of Death (BSOD) crashes.…
If you like to separate your workflow onto multiple monitors but hate the gap and bezel between screens, Dell’s new display was made for you. Announced on Tuesday at CES, the Dell UltraSharp 52 (U5226KW) offers 52 inches of 6K resolution screen real estate that you can divide into up to four virtual monitors, supporting input either from up to four different devices, or one computer that creates that many desktops.…
As 2025 is drawing to an end we want to express our gratitude to our Members, staff, and community for the immense work accomplished this year towards making the web work — for everyone! We wish you all the best for 2026.
This year we made an animated holiday greeting. Warmly, from all of us at the World Wide Web Consortium.
In accordance with the W3C Patent Policy, W3C has launched a Second Screen Working Group Patent Advisory Group (PAG) in response to disclosures and exclusions related to the Open Screen Application Protocol specification of the Second Screen Working Group; see the PAG charter. W3C launches a PAG to resolve issues in the event a patent has been disclosed that may be essential, but is not available under the W3C Royalty-Free licensing requirements. Public comments regarding these disclosures may be sent to public-secondscreen-pag@w3.org (public archive). Learn more about Patent Advisory Groups.